You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "veghlaci05 (via GitHub)" <gi...@apache.org> on 2023/08/08 13:41:29 UTC

[GitHub] [hive] veghlaci05 opened a new pull request, #4566: HIVE-27481: TxnHandler cleanup

veghlaci05 opened a new pull request, #4566:
URL: https://github.com/apache/hive/pull/4566

   ### What changes were proposed in this pull request?
   Standardization and simplification of JDBC related code inTxnHandler
   
   Key changes:
   
       Standardized transaction handling
       Standardized Retry and exception handling
       No direct connection handling
       Use of RowMapper/ResultSetExtractor/RowCallbackHandler interfaces for DB <-> DAO class mapping
       Extract code parts from TxnHandler: Ideally these classes should contain the business logic only, and the infrastructure related code parts (transaction, connection, retry handling, object mapping, etc) should be extracted into individual classes.
   
   ### Why are the changes needed?
   TxnHandler is getting too complex, and a source of tech-debt accumulation
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### Is the change a dependency upgrade?
   No
   
   ### How was this patch tested?
   Through unit tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418925637


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/MultiDataSourceJdbcResource.java:
##########
@@ -46,19 +57,23 @@ public class MultiDataSourceJdbcResource {
 
   private static final Logger LOG = LoggerFactory.getLogger(MultiDataSourceJdbcResource.class);
 
-  private final ThreadLocal<ContextNode<String>> threadLocal = new ThreadLocal<>();
+  private final StackThreadLocal<String> threadLocal = new StackThreadLocal<>();

Review Comment:
   ok, i can see it's used in SqlRetryHandler as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418958428


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -470,28 +320,29 @@ private boolean checkIfTableIsUsable(String tableName, boolean configValue) thro
       // don't check it if disabled
       return false;
     }
-    Connection dbConn = null;
-    boolean tableExists = true;
+    jdbcResource.bindDataSource(POOL_TX);
     try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      try (Statement stmt = dbConn.createStatement()) {
-        // Dummy query to see if table exists
-        try (ResultSet rs = stmt.executeQuery("SELECT 1 FROM \"" + tableName + "\"")) {
-          rs.next();
-        }
-      }
-    } catch (SQLException e) {
+      jdbcResource.getJdbcTemplate().query("SELECT 1 FROM \"" + tableName + "\"", new MapSqlParameterSource(),
+          rs -> {
+            if (rs.next()) {
+              return rs.getInt(1);
+            }
+            return -1;

Review Comment:
   -1 would be cast to false?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418812037


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -546,61 +518,28 @@ public long findMinOpenTxnIdForCleaner() throws MetaException {
   @RetrySemantics.Idempotent
   @Deprecated
   public long findMinTxnIdSeenOpen() {
-    if (!useMinHistoryLevel || useMinHistoryWriteId) {
+    if (!TxnHandlingFeatures.useMinHistoryLevel() || TxnHandlingFeatures.useMinHistoryWriteId()) {

Review Comment:
   ````
   TxnHandler.ConfVars
       .useMinHistoryLevel()
       . setUseMinHistoryLevel(boolean toggle)
       .useMinHistoryWriteId()
   ````



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -546,61 +518,28 @@ public long findMinOpenTxnIdForCleaner() throws MetaException {
   @RetrySemantics.Idempotent
   @Deprecated
   public long findMinTxnIdSeenOpen() {
-    if (!useMinHistoryLevel || useMinHistoryWriteId) {
+    if (!TxnHandlingFeatures.useMinHistoryLevel() || TxnHandlingFeatures.useMinHistoryWriteId()) {

Review Comment:
   ````
   TxnHandler.ConfVars
       .useMinHistoryLevel()
       .setUseMinHistoryLevel(boolean toggle)
       .useMinHistoryWriteId()
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418842387


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          } else if (!context.isCompleted()) {
+            jdbcResource.getTransactionManager().commit(context);
+          }
           return result;
+        } catch (ProgrammaticRollbackException e) {
+          if (context != null && !context.isCompleted()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          }          
+          return e.getResult();
         } catch (Exception e) {
-          if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
-              Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
-            throw e;
-          }
           if (context != null) {
-            if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {
+            if (transactional.noRollbackFor().length > 0 || transactional.noRollbackForClassName().length > 0) {
+              if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
+                  Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
+                jdbcResource.getTransactionManager().commit(context);
+              } else {
+                jdbcResource.getTransactionManager().rollback(context);
+              }
+            } else if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {

Review Comment:
   yea, sorry, removed
   is there an order to check for `noRollbackFor` and `rollbackFor`? could we get different outcome?
   is that possible to extract this logic into a method `shouldRollback()` or somethiing?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418852289


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import java.util.Stack;
+
+public class StackThreadLocal<T> {

Review Comment:
   Where should I move it? There are several util classes across the project, not sure where it belongs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419182862


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/retryhandling/SqlRetryHandler.java:
##########
@@ -123,32 +124,29 @@ public <Result> Result executeWithRetry(SqlRetryCallProperties properties, SqlRe
       if (properties.isLockInternally()) {
         lockInternal();
       }
-      threadLocal.set(new ContextNode<>(threadLocal.get(), new Object()));
+      threadLocal.set(new Object());

Review Comment:
   but why we need to track this is we don't care what is inside. Can't we just have counter + -



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418834695


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          } else if (!context.isCompleted()) {
+            jdbcResource.getTransactionManager().commit(context);
+          }
           return result;
+        } catch (ProgrammaticRollbackException e) {
+          if (context != null && !context.isCompleted()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          }          
+          return e.getResult();
         } catch (Exception e) {
-          if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
-              Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
-            throw e;
-          }
           if (context != null) {
-            if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {
+            if (transactional.noRollbackFor().length > 0 || transactional.noRollbackForClassName().length > 0) {
+              if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
+                  Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
+                jdbcResource.getTransactionManager().commit(context);
+              } else {
+                jdbcResource.getTransactionManager().rollback(context);
+              }
+            } else if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {

Review Comment:
   !! it's the same if predicate as above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418938461


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +181,115 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockHandler txnLockHandler;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf,
+                  MetastoreConf.getIntVar(conf, ConfVars.HIVE_COMPACTOR_CONNECTION_POOLING_MAX_CONNECTIONS));
+            }            
+          }
+          if (dbProduct == null) {
+            try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPool)) {
+              determineDatabaseProduct(dbConn);
+            } catch (SQLException e) {
+              LOG.error("Unable to determine database product", e);
+              throw new RuntimeException(e);
+            }
+          }
+          if (sqlGenerator == null) {
+            sqlGenerator = new SQLGenerator(dbProduct, conf);
+          }
+          initialized = true;
         }
       }
+    }
 
-      if (sqlGenerator == null) {
-        sqlGenerator = new SQLGenerator(dbProduct, conf);
-      }
-      
-      if (jdbcResource == null) {
-        jdbcResource = new MultiDataSourceJdbcResource(dbProduct);
-        jdbcResource.registerDataSource(POOL_TX, connPool);
-        jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
-      }      
+    if (jdbcResource == null) {
+      jdbcResource = new MultiDataSourceJdbcResource(dbProduct, conf, sqlGenerator);
+      jdbcResource.registerDataSource(POOL_TX, connPool);
+      jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
+      jdbcResource.registerDataSource(POOL_COMPACTOR, connPoolCompactor);
     }
+    
+    mutexAPI = new HiveMutex(sqlGenerator, jdbcResource);
 
     numOpenTxns = Metrics.getOrCreateGauge(MetricsConstants.NUM_OPEN_TXNS);
 
     timeout = MetastoreConf.getTimeVar(conf, ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
     replicationTxnTimeout = MetastoreConf.getTimeVar(conf, ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    retryInterval = MetastoreConf.getTimeVar(conf, ConfVars.HMS_HANDLER_INTERVAL,
-        TimeUnit.MILLISECONDS);
-    retryLimit = MetastoreConf.getIntVar(conf, ConfVars.HMS_HANDLER_ATTEMPTS);
-    deadlockRetryInterval = retryInterval / 10;
     maxOpenTxns = MetastoreConf.getIntVar(conf, ConfVars.MAX_OPEN_TXNS);
-    maxBatchSize = MetastoreConf.getIntVar(conf, ConfVars.JDBC_MAX_BATCH_SIZE);
-
     openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
     
     try {
-      useMinHistoryWriteId = checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
-      
-      // override the config if table does not exists anymore
+      TxnHandlingFeatures.setUseMinHistoryWriteId(checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
+              MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))

Review Comment:
   formatting, extra spaces
   btw, can't we move that logic checkIfTableIsUsable inside of `TxnHandler.ConfVars` init or something. Another option do lazy evaluation in `TxnHandler.ConfVars` getter



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1750519670

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [10 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [119 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "zratkai (via GitHub)" <gi...@apache.org>.
zratkai commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1406030830


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -193,10 +193,17 @@ public boolean isDeadlock(SQLException e) {
 
   /**
    * Is the given exception a table not found exception
-   * @param e Exception
+   * @param ex Exception
    * @return
    */
-  public boolean isTableNotExistsError(SQLException e) {
+  public boolean isTableNotExistsError(Throwable ex) {
+    while (ex != null && !(ex instanceof SQLException) ) {
+      ex = ex.getCause();
+    }
+    if (ex == null) {
+      throw new IllegalArgumentException("No SQLException found in the exception chain!");
+    }
+    SQLException e = (SQLException) ex;    

Review Comment:
   Before casting it would be good to check if it is a SQLException.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418953054


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -546,41 +548,42 @@ public boolean supportsGetGeneratedKeys() throws MetaException {
     }
   }
 
-  public boolean isDuplicateKeyError(SQLException ex) {
+  public boolean isDuplicateKeyError(Throwable ex) {

Review Comment:
   i would just change input param to `t`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419049069


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {
+    int threshold = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD);
+    MetricsInfo metrics = jdbcResource.execute(MetricsInfoHandler.INSTANCE);
+    Set<String> resourceNames = jdbcResource.execute(new TablesWithAbortedTxnsHandler(threshold));
+    metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
+    metrics.setTablesWithXAbortedTxns(resourceNames);    
+    return metrics;
   }
 
-  /**
-   * Heartbeats on the lock table.  This commits, so do not enter it with any state.
-   * Should not be called on a lock that belongs to transaction.
-   */
-  private void heartbeatLock(Connection dbConn, long extLockId)
-    throws NoSuchLockException, SQLException, MetaException {
-    // If the lock id is 0, then there are no locks in this heartbeat
-    if (extLockId == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String updateHeartbeatQuery = "UPDATE \"HIVE_LOCKS\" SET \"HL_LAST_HEARTBEAT\" = " +
-          getEpochFn(dbProduct) + " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute update <{}>", updateHeartbeatQuery);
-      int rc = stmt.executeUpdate(updateHeartbeatQuery);
-      if (rc < 1) {
-        LOG.error("Failure to update last heartbeat for extLockId={}.", extLockId);
-        dbConn.rollback();
-        throw new NoSuchLockException("No such lock: " + JavaUtils.lockIdToString(extLockId));
-      }
-      LOG.debug("Successfully heartbeated for extLockId={}", extLockId);
-      dbConn.commit();
-    }
-  }
 
-  // Heartbeats on the txn table.  This commits, so do not enter it with any state
-  private void heartbeatTxn(Connection dbConn, long txnid)
-    throws NoSuchTxnException, TxnAbortedException, SQLException, MetaException {
-    // If the txnid is 0, then there are no transactions in this heartbeat
-    if (txnid == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String s = "UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-          " WHERE \"TXN_ID\" = " + txnid + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-      LOG.debug("Going to execute update <{}>", s);
-      int rc = stmt.executeUpdate(s);
-      if (rc < 1) {
-        ensureValidTxn(dbConn, txnid, stmt); // This should now throw some useful exception.
-        LOG.error("Can neither heartbeat txn (txnId={}) nor confirm it as invalid.", txnid);
-        dbConn.rollback();
-        throw new NoSuchTxnException("No such txn: " + txnid);
-      }
-      LOG.debug("Successfully heartbeated for txnId={}", txnid);
-      dbConn.commit();
-    }
+  private static void shouldNeverHappen(long txnid) {
+    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
   }
 
   /**
-   * Returns the state of the transaction if it's able to determine it. Some cases where it cannot:
-   * 1. txnid was Aborted/Committed and then GC'd (compacted)
-   * 2. txnid was committed but it didn't modify anything (nothing in COMPLETED_TXN_COMPONENTS)
+   * Retry-by-caller note:
+   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
+   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
    */
-  private TxnStatus findTxnState(long txnid, Statement stmt) throws SQLException, MetaException {
-    String s = "SELECT \"TXN_STATE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnid;
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        s =
-            sqlGenerator.addLimitClause(1, "1 FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_TXNID\" = "
-                + txnid);
-        LOG.debug("Going to execute query <{}>", s);
-        try (ResultSet rs2 = stmt.executeQuery(s)) {
-          if (rs2.next()) {
-            return TxnStatus.COMMITTED;
-          }
-        }
-        // could also check WRITE_SET but that seems overkill
-        return TxnStatus.UNKNOWN;
-      }
-      return TxnStatus.fromString(rs.getString(1));
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void addDynamicPartitions(AddDynamicPartitions rqst) throws NoSuchTxnException,  TxnAbortedException, MetaException {
+    TxnType txnType = jdbcResource.execute(new GetOpenTxnTypeAndLockHandler(sqlGenerator, rqst.getTxnid()));
+    if (txnType == null) {
+      //ensures txn is still there and in expected state
+      new EnsureValidTxnFunction(rqst.getTxnid()).execute(jdbcResource);
+      shouldNeverHappen(rqst.getTxnid());
+    }
+    jdbcResource.execute(new InsertTxnComponentsCommand(rqst));
+    jdbcResource.getJdbcTemplate().update("DELETE FROM \"TXN_COMPONENTS\" " +
+            "WHERE \"TC_TXNID\" = :txnId AND \"TC_DATABASE\" = :dbName AND \"TC_TABLE\" = :tableName AND \"TC_PARTITION\" IS NULL",
+        new MapSqlParameterSource()
+            .addValue("txnId", rqst.getTxnid())
+            .addValue("dbName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getDbname()))
+            .addValue("tableName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getTablename())));
   }
 
   /**
-   * Checks if all the txns in the list are in open state and not read-only.
-   * @param txnIds list of txns to be evaluated for open state/read-only status
-   * @param stmt db statement
-   * @return If all the txns in open state and not read-only, then return true else false
+   * Clean up corresponding records in metastore tables when corresponding object is dropped,
+   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
+   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
+   * operations.
+   * <p>
+   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
    */
-  private boolean isTxnsOpenAndNotReadOnly(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Get the count of txns from the given list that are in open state and not read-only.
-    // If the returned count is same as the input number of txns, then all txns are in open state and not read-only.
-    prefix.append("SELECT COUNT(*) FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN
-        + " AND \"TXN_TYPE\" != " + TxnType.READ_ONLY.getValue() + " AND ");
-
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-
-    long count = 0;
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        if (rs.next()) {
-          count += rs.getLong(1);
-        }
-      }
-    }
-    return count == txnIds.size();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), keepTxnToWriteIdMetaData, null)
+        .execute(jdbcResource);
   }
 
-  /**
-   * Get txns from the list that are either aborted or read-only.
-   * @param txnIds list of txns to be evaluated for aborted state/read-only status
-   * @param stmt db statement
-   */
-  private String getAbortedAndReadOnlyTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are either aborted or read-only.
-    prefix.append("SELECT \"TXN_ID\", \"TXN_STATE\", \"TXN_TYPE\" FROM \"TXNS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          TxnStatus txnState = TxnStatus.fromString(rs.getString(2));
-          TxnType txnType = TxnType.findByValue(rs.getInt(3));
-
-          if (txnState != TxnStatus.OPEN) {
-            txnInfo.append("{").append(txnId).append(",").append(txnState).append("}");
-          } else if (txnType == TxnType.READ_ONLY) {
-            txnInfo.append("{").append(txnId).append(",read-only}");
-          }
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), false, txnId)
+        .execute(jdbcResource);
   }
-
+  
   /**
-   * Get txns from the list that are committed.
-   * @param txnIds list of txns to be evaluated for committed state
-   * @param stmt db statement
+   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
    */
-  private String getCommittedTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are committed.
-    prefix.append("SELECT \"CTC_TXNID\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"CTC_TXNID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          txnInfo.append("{").append(txnId).append(",c}");
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
+      String newCatName, String newDbName, String newTabName, String newPartName)
+      throws MetaException {
+    new OnRenameFunction(oldCatName, oldDbName, oldTabName, oldPartName, newCatName, newDbName, newTabName, newPartName).execute(jdbcResource);
   }
-
   /**
-   * Used to raise an informative error when the caller expected a txn in a particular TxnStatus
-   * but found it in some other status
+   * For testing only, do not use.
    */
-  private static void raiseTxnUnexpectedState(TxnStatus actualStatus, long txnid)
-    throws NoSuchTxnException, TxnAbortedException {
-    switch (actualStatus) {
-      case ABORTED:
-        throw new TxnAbortedException("Transaction " + JavaUtils.txnIdToString(txnid) + " already aborted");
-      case COMMITTED:
-        throw new NoSuchTxnException("Transaction " + JavaUtils.txnIdToString(txnid) + " is already committed.");
-      case UNKNOWN:
-        throw new NoSuchTxnException("No such transaction " + JavaUtils.txnIdToString(txnid));
-      case OPEN:
-        throw new NoSuchTxnException(JavaUtils.txnIdToString(txnid) + " is " + TxnStatus.OPEN);
-      default:
-        throw new IllegalArgumentException("Unknown TxnStatus " + actualStatus);
-    }
+  @VisibleForTesting
+  public int numLocksInLockTable() {

Review Comment:
   should we rename to `getNumLocks()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419071876


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -6052,450 +1096,53 @@ protected synchronized static DataSource setupJdbcConnectionPool(Configuration c
         throw new RuntimeException(e);
       }
     } else {
-      String connectionPooler = MetastoreConf.getVar(conf, ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
+      String connectionPooler = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
       if ("none".equals(connectionPooler)) {
         LOG.info("Choosing not to pool JDBC connections");
-        return new NoPoolConnectionPool(conf);
+        return new NoPoolConnectionPool(conf, dbProduct);
       } else {
         throw new RuntimeException("Unknown JDBC connection pooling " + connectionPooler);
       }
     }
   }
 
-  /**
-   * Returns true if {@code ex} should be retried
-   */
-  static boolean isRetryable(Configuration conf, Exception ex) {
-    if(ex instanceof SQLException) {
-      SQLException sqlException = (SQLException)ex;
-      if (MANUAL_RETRY.equalsIgnoreCase(sqlException.getSQLState())) {
-        // Manual retry exception was thrown
-        return true;
-      }
-      if ("08S01".equalsIgnoreCase(sqlException.getSQLState())) {
-        //in MSSQL this means Communication Link Failure
-        return true;
-      }
-      if ("ORA-08176".equalsIgnoreCase(sqlException.getSQLState()) ||
-        sqlException.getMessage().contains("consistent read failure; rollback data not available")) {
-        return true;
-      }
-
-      String regex = MetastoreConf.getVar(conf, ConfVars.TXN_RETRYABLE_SQLEX_REGEX);
-      if (regex != null && !regex.isEmpty()) {
-        String[] patterns = regex.split(",(?=\\S)");
-        String message = getMessage((SQLException)ex);
-        for (String p : patterns) {
-          if (Pattern.matches(p, message)) {
-            return true;
-          }
-        }
-      }
-      //see also https://issues.apache.org/jira/browse/HIVE-9938
-    }
-    return false;
-  }
-
-  private boolean isDuplicateKeyError(SQLException ex) {
-    return dbProduct.isDuplicateKeyError(ex);
-  }
-
-  private static String getMessage(SQLException ex) {
-    return ex.getMessage() + " (SQLState=" + ex.getSQLState() + ", ErrorCode=" + ex.getErrorCode() + ")";
-  }
-  static String quoteString(String input) {
-    return "'" + input + "'";
-  }
-  static String quoteChar(char c) {
-    return "'" + c + "'";
-  }
-
-  /**
-   * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
-   * Select ... For Update to sequence operations properly.  In practice that means when running
-   * with Derby database.  See more notes at class level.
-   */
-  protected void lockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.lock();
-    }
-  }
-  protected void unlockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.unlock();
-    }
-  }
   @Override
   @RetrySemantics.Idempotent
   public MutexAPI getMutexAPI() {
-    return this;
+    return mutexAPI;
   }
 
   @Override
   public LockHandle acquireLock(String key) throws MetaException {
-    /**
-     * The implementation here is a bit kludgey but done so that code exercised by unit tests
-     * (which run against Derby which has no support for select for update) is as similar to
-     * production code as possible.
-     * In particular, with Derby we always run in a single process with a single metastore and
-     * the absence of For Update is handled via a Semaphore.  The later would strictly speaking
-     * make the SQL statements below unnecessary (for Derby), but then they would not be tested.
-     */
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    boolean needToCloseConn = true;
-    try {
-      try {
-        String sqlStmt = sqlGenerator.addForUpdateClause("SELECT \"MT_COMMENT\", \"MT_KEY2\" FROM \"AUX_TABLE\" WHERE \"MT_KEY1\"=" + quoteString(key));
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPoolMutex);
-        stmt = dbConn.createStatement();
-        LOG.debug("About to execute SQL: {}", sqlStmt);
-        rs = stmt.executeQuery(sqlStmt);
-        if (!rs.next()) {
-          close(rs);
-          try {
-            stmt.executeUpdate("INSERT INTO \"AUX_TABLE\" (\"MT_KEY1\", \"MT_KEY2\") VALUES(" + quoteString(key) + ", 0)");
-            dbConn.commit();
-          } catch (SQLException ex) {
-            if (!isDuplicateKeyError(ex)) {
-              throw new RuntimeException("Unable to lock " + quoteString(key) + " due to: " + getMessage(ex), ex);
-            }
-            //if here, it means a concrurrent acquireLock() inserted the 'key'
-
-            //rollback is done for the benefit of Postgres which throws (SQLState=25P02, ErrorCode=0) if
-            //you attempt any stmt in a txn which had an error.
-            dbConn.rollback();
-          }
-          rs = stmt.executeQuery(sqlStmt);
-          if (!rs.next()) {
-            throw new IllegalStateException("Unable to lock " + quoteString(key) + ".  Expected row in AUX_TABLE is missing.");
-          }
-        }
-        Semaphore derbySemaphore = null;
-        if(dbProduct.isDERBY()) {
-          derbyKey2Lock.putIfAbsent(key, new Semaphore(1));
-          derbySemaphore =  derbyKey2Lock.get(key);
-          derbySemaphore.acquire();
-        }
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("{} locked by {}", quoteString(key), quoteString(TxnHandler.hostname));
-        }
-        needToCloseConn = false;  //The connection is good, we need not close it
-        //OK, so now we have a lock
-        return new LockHandleImpl(dbConn, stmt, rs, key, derbySemaphore);
-      } catch (SQLException ex) {
-        checkRetryable(ex, "acquireLock(" + key + ")");
-        throw new MetaException("Unable to lock " + quoteString(key) + " due to: " + getMessage(ex) + "; " + StringUtils.stringifyException(ex));
-      }
-      catch(InterruptedException ex) {
-        throw new MetaException("Unable to lock " + quoteString(key) + " due to: " + ex.getMessage() + StringUtils.stringifyException(ex));
-      }
-      finally {
-        if (needToCloseConn) {
-          rollbackDBConn(dbConn);
-          close(rs, stmt, dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException ex) {
-      return acquireLock(key);
-    }
+    return mutexAPI.acquireLock(key);
   }
 
   @Override
-  public void acquireLock(String key, LockHandle handle) {
-    //the idea is that this will use LockHandle.dbConn
-    throw new NotImplementedException("acquireLock(String, LockHandle) is not implemented");
+  public void acquireLock(String key, LockHandle handle) throws MetaException {
+    mutexAPI.acquireLock(key, handle);
   }
 
   /**
    * Acquire the global txn lock, used to mutex the openTxn and commitTxn.
    * @param shared either SHARED_READ or EXCLUSIVE
-   * @throws SQLException
    */
-  private void acquireTxnLock(Statement stmt, boolean shared) throws SQLException, MetaException {
+  private void acquireTxnLock(boolean shared) throws MetaException {
     String sqlStmt = sqlGenerator.createTxnLockStatement(shared);
-    stmt.execute(sqlStmt);
-    LOG.debug("TXN lock locked by {} in mode {}", quoteString(TxnHandler.hostname), shared);
-  }
-
-  private static final class LockHandleImpl implements LockHandle {
-    private final Connection dbConn;
-    private final Statement stmt;
-    private final ResultSet rs;
-    private final Semaphore derbySemaphore;
-    private final String key;
-    private final Long lastUpdateTime;
-
-    LockHandleImpl(Connection conn, Statement stmt, ResultSet rs, String key, Semaphore derbySemaphore) {
-      this.dbConn = conn;
-      this.stmt = stmt;
-      this.rs = rs;
-      this.derbySemaphore = derbySemaphore;
-      if(derbySemaphore != null) {
-        //oterwise it may later release permit acquired by someone else
-        assert derbySemaphore.availablePermits() == 0 : "Expected locked Semaphore";
-      }
-      this.key = key;
-      Long lastUpdateTime;
-      try {
-        lastUpdateTime = rs.getLong("MT_KEY2");
-      } catch (SQLException e) {
-        LOG.warn("Couldn't resolve MT_KEY2 for MT_KEY1=" + quoteString(this.key), e);
-        lastUpdateTime = -1L;
-      }
-      this.lastUpdateTime = lastUpdateTime;
-    }
-
-    @Override
-    public void releaseLocks() {
-      rollbackDBConn(dbConn);
-      TxnHandler.close(rs, stmt, dbConn);
-      if(derbySemaphore != null) {
-        derbySemaphore.release();
-      }
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("{} unlocked by {}", quoteString(key), quoteString(TxnHandler.hostname));
-      }
-    }
-
-    @Override
-    public Long getLastUpdateTime() {
-      return lastUpdateTime;
-    }
-
-    @Override
-    public void releaseLocks(Long timestamp) {
-      try {
-        stmt.executeUpdate("UPDATE \"AUX_TABLE\" SET \"MT_KEY2\" = "+ timestamp + " WHERE \"MT_KEY1\"=" + quoteString(key));
-        dbConn.commit();
-      } catch (SQLException ex) {
-        LOG.warn("Unable to update MT_KEY2 value for MT_KEY1=" + key, ex);
-        rollbackDBConn(dbConn);
-      }
-      TxnHandler.close(rs, stmt, dbConn);
-      if(derbySemaphore != null) {
-        derbySemaphore.release();
-      }
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("{} unlocked by {}", quoteString(key), quoteString(TxnHandler.hostname));
-      }
-    }
-
-    @Override
-    public void close() {
-      releaseLocks();
-    }
-
-  }
-
-
-  private static class NoPoolConnectionPool implements DataSource {
-    // Note that this depends on the fact that no-one in this class calls anything but
-    // getConnection.  If you want to use any of the Logger or wrap calls you'll have to
-    // implement them.
-    private final Configuration conf;
-    private Driver driver;
-    private String connString;
-    private String user;
-    private String passwd;
-
-    public NoPoolConnectionPool(Configuration conf) {
-      this.conf = conf;
-    }
-
-    @Override
-    public Connection getConnection() throws SQLException {
-      if (user == null) {
-        user = DataSourceProvider.getMetastoreJdbcUser(conf);
-        passwd = DataSourceProvider.getMetastoreJdbcPasswd(conf);
-      }
-      return getConnection(user, passwd);
-    }
-
-    @Override
-    public Connection getConnection(String username, String password) throws SQLException {
-      // Find the JDBC driver
-      if (driver == null) {
-        String driverName = MetastoreConf.getVar(conf, ConfVars.CONNECTION_DRIVER);
-        if (driverName == null || driverName.equals("")) {
-          String msg = "JDBC driver for transaction db not set in configuration " +
-              "file, need to set " + ConfVars.CONNECTION_DRIVER.getVarname();
-          LOG.error(msg);
-          throw new RuntimeException(msg);
-        }
-        try {
-          LOG.info("Going to load JDBC driver {}", driverName);
-          driver = (Driver) Class.forName(driverName).newInstance();
-        } catch (InstantiationException e) {
-          throw new RuntimeException("Unable to instantiate driver " + driverName + ", " +
-              e.getMessage(), e);
-        } catch (IllegalAccessException e) {
-          throw new RuntimeException(
-              "Unable to access driver " + driverName + ", " + e.getMessage(),
-              e);
-        } catch (ClassNotFoundException e) {
-          throw new RuntimeException("Unable to find driver " + driverName + ", " + e.getMessage(),
-              e);
-        }
-        connString = MetastoreConf.getVar(conf, ConfVars.CONNECT_URL_KEY);
-      }
-
-      try {
-        LOG.info("Connecting to transaction db with connection string {}", connString);
-        Properties connectionProps = new Properties();
-        connectionProps.setProperty("user", username);
-        connectionProps.setProperty("password", password);
-        Connection conn = driver.connect(connString, connectionProps);
-        String prepareStmt = dbProduct != null ? dbProduct.getPrepareTxnStmt() : null;
-        if (prepareStmt != null) {
-          try (Statement stmt = conn.createStatement()) {
-            stmt.execute(prepareStmt);
-          }
-        }
-        conn.setAutoCommit(false);
-        return conn;
-      } catch (SQLException e) {
-        throw new RuntimeException("Unable to connect to transaction manager using " + connString
-            + ", " + e.getMessage(), e);
-      }
-    }
-
-    @Override
-    public PrintWriter getLogWriter() throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public void setLogWriter(PrintWriter out) throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public void setLoginTimeout(int seconds) throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public int getLoginTimeout() throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public java.util.logging.Logger getParentLogger() throws SQLFeatureNotSupportedException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public <T> T unwrap(Class<T> iface) throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public boolean isWrapperFor(Class<?> iface) throws SQLException {
-      throw new UnsupportedOperationException();
-    }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> {
+        stmt.execute(sqlStmt);
+        return null;
+    });
+    LOG.debug("TXN lock locked by '{}' in mode {}", TxnHandler.hostname, shared);
   }
 
   @Override
   @RetrySemantics.SafeToRetry

Review Comment:
   should we define retry semantic here, as it's also present in interface?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419107740


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +182,141 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+  public static class TxnHandlingFeatures {    

Review Comment:
   please rename to ConfVars. 
   ````
   private static final ConfVars confVars;
   
   confVars = new ConfVars(table->checkIfTableIsUsable(table), conf)
   
   public static final ConfVars {
     private boolean useMinHistoryLevel;
     private boolean useMinHistoryWriteId;
     
     public ConfVars(Predicate<String> tableCheck, Conf conf){
       useMinHistoryLevel = tableCheck.test("MIN_HISTORY_WRITE_ID", 
     MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))
     }
   }
   ````
   
   getConfVars().useMinHistoryWriteId
   getConfVars().useMinHistoryLevel



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419171369


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/commands/AddWriteIdsToMinHistoryCommand.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.commands;

Review Comment:
   txn/jdbc, it's under root now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419036479


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +431,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table
+    StopWatch generateTransactionWatch = new StopWatch();
+    generateTransactionWatch.start();
 
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        generateTransactionWatch.stop();
-        long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
-        TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-        if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
-          /*
-           * The commit was too slow, we can not allow this to continue (except if it is read only,
-           * since that can not cause dirty reads).
-           * When calculating the snapshot for a given transaction, we look back for possible open transactions
-           * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
-           * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
-           * because there can be other transactions running, that didn't considered this transactionId open,
-           * this could cause dirty reads.
-           */
-          LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
-          deleteInvalidOpenTransactions(dbConn, txnIds);
-          dbConn.commit();
-          /*
-           * We do not throw RetryException directly, to not circumvent the max retry limit
-           */
-          throw new SQLException("OpenTxnTimeOut exceeded", MANUAL_RETRY);
-        }
-        return new OpenTxnsResponse(txnIds);
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "openTxns(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return openTxns(rqst);
-    }
-  }
+    List<Long> txnIds = new OpenTxnsFunction(rqst, openTxnTimeOutMillis, transactionalListeners).execute(jdbcResource);
 
-  private List<Long> openTxns(Connection dbConn, OpenTxnRequest rqst)
-          throws SQLException, MetaException {
-    int numTxns = rqst.getNum_txns();
-    // Make sure the user has not requested an insane amount of txns.
-    int maxTxns = MetastoreConf.getIntVar(conf, ConfVars.TXN_MAX_OPEN_BATCH);
-    if (numTxns > maxTxns) {
-      numTxns = maxTxns;
-    }
-    List<PreparedStatement> insertPreparedStmts = null;
+    LOG.debug("Going to commit");
+    jdbcResource.getTransactionManager().getActiveTransaction().createSavepoint();

Review Comment:
   why do we need this, it's not present in original code?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419080698


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {

Review Comment:
   missing @Override



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418808055


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -193,10 +194,11 @@ public boolean isDeadlock(SQLException e) {
 
   /**
    * Is the given exception a table not found exception
-   * @param e Exception
+   * @param ex Exception
    * @return
    */
-  public boolean isTableNotExistsError(SQLException e) {
+  public boolean isTableNotExistsError(Throwable ex) {

Review Comment:
   The logic slightly changed: Now the method accepts any kind of exceptions and finds an `SQLException` within the exception chain. If no `SQLException` found an `IllegalArgumentException` is thrown. This makes the caller code slightly less complex, as there is no need to find the `SQLException` if it's not the top one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422557800


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +182,141 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+  public static class TxnHandlingFeatures {    

Review Comment:
   done, except I did not remove the getters as they are used outside of `TxnHandler` too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422535425


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -5785,261 +1068,22 @@ private void timeOutLocks() {
    */
   @RetrySemantics.Idempotent
   public void performTimeOuts() {
-    jdbcResource.bindDataSource(POOL_TX);
-    try (TransactionContext context = jdbcResource.getTransactionManager().getTransaction(PROPAGATION_REQUIRED)) {
-      //We currently commit after selecting the TXNS to abort.  So whether SERIALIZABLE
-      //READ_COMMITTED, the effect is the same.  We could use FOR UPDATE on Select from TXNS
-      //and do the whole performTimeOuts() in a single huge transaction, but the only benefit
-      //would be to make sure someone cannot heartbeat one of these txns at the same time.
-      //The attempt to heartbeat would block and fail immediately after it's unblocked.
-      //With current (RC + multiple txns) implementation it is possible for someone to send
-      //heartbeat at the very end of the expiry interval, and just after the Select from TXNS
-      //is made, in which case heartbeat will succeed but txn will still be Aborted.
-      //Solving this corner case is not worth the perf penalty.  The client should heartbeat in a
-      //timely way.
-      timeOutLocks();
-      while (true) {
-        String s = " \"TXN_ID\" FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN +
-            " AND (" +
-            "\"TXN_TYPE\" != " + TxnType.REPL_CREATED.getValue() +
-            " AND \"TXN_LAST_HEARTBEAT\" <  " + getEpochFn(dbProduct) + "-" + timeout +
-            " OR " +
-            " \"TXN_TYPE\" = " + TxnType.REPL_CREATED.getValue() +
-            " AND \"TXN_LAST_HEARTBEAT\" <  " + getEpochFn(dbProduct) + "-" + replicationTxnTimeout +
-            ")";
-        //safety valve for extreme cases
-        s = sqlGenerator.addLimitClause(10 * TIMED_OUT_TXN_ABORT_BATCH_SIZE, s);
-
-        LOG.debug("Going to execute query <{}>", s);
-        List<List<Long>> timedOutTxns = jdbcResource.getJdbcTemplate().query(s, rs -> {
-          List<List<Long>> txnbatch = new ArrayList<>();
-          List<Long> currentBatch = new ArrayList<>(TIMED_OUT_TXN_ABORT_BATCH_SIZE);
-          while (rs.next()) {
-            currentBatch.add(rs.getLong(1));
-            if (currentBatch.size() == TIMED_OUT_TXN_ABORT_BATCH_SIZE) {
-              txnbatch.add(currentBatch);
-              currentBatch = new ArrayList<>(TIMED_OUT_TXN_ABORT_BATCH_SIZE);
-            }
-          }
-          if (currentBatch.size() > 0) {
-            txnbatch.add(currentBatch);
-          }
-          return txnbatch;
-        });
-        //noinspection DataFlowIssue
-        if (timedOutTxns.size() == 0) {
-          jdbcResource.getTransactionManager().commit(context);
-          return;
-        }
-
-        Object savePoint = context.getTransactionStatus().createSavepoint();
-
-        int numTxnsAborted = 0;
-        for (List<Long> batchToAbort : timedOutTxns) {
-          context.getTransactionStatus().releaseSavepoint(savePoint);
-          savePoint = context.getTransactionStatus().createSavepoint();
-          if (abortTxns(jdbcResource.getConnection(), batchToAbort, true, false, false, TxnErrorMsg.ABORT_TIMEOUT) == batchToAbort.size()) {
-            numTxnsAborted += batchToAbort.size();
-            //todo: add TXNS.COMMENT filed and set it to 'aborted by system due to timeout'
-          } else {
-            //could not abort all txns in this batch - this may happen because in parallel with this
-            //operation there was activity on one of the txns in this batch (commit/abort/heartbeat)
-            //This is not likely but may happen if client experiences long pause between heartbeats or
-            //unusually long/extreme pauses between heartbeat() calls and other logic in checkLock(),
-            //lock(), etc.
-            context.getTransactionStatus().rollbackToSavepoint(savePoint);
-          }
-        }
-        LOG.info("Aborted {} transaction(s) due to timeout", numTxnsAborted);
-        if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-          Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_TIMED_OUT_TXNS).inc(numTxnsAborted);
-        }
-      }
-    } catch (MetaException | SQLException e) {
-      LOG.warn("Aborting timed out transactions failed due to " + e.getMessage(), e);
-    } finally {
-      jdbcResource.unbindDataSource();
-    }
+    new PerformTimeoutsFunction(timeout, replicationTxnTimeout).execute(jdbcResource);
   }
 
   @Override
   @RetrySemantics.ReadOnly
   public void countOpenTxns() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        String s = "SELECT COUNT(*) FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN;
-        LOG.debug("Going to execute query <{}>", s);
-        rs = stmt.executeQuery(s);
-        if (!rs.next()) {
-          LOG.error("Transaction database not properly configured, can't find txn_state from TXNS.");
-        } else {
-          Long numOpen = rs.getLong(1);
-          if (numOpen > Integer.MAX_VALUE) {
-            LOG.error("Open transaction count above {}, can't count that high!", Integer.MAX_VALUE);
-          } else {
-            numOpenTxns.set(numOpen.intValue());
-          }
-        }
-      } catch (SQLException e) {
-        LOG.info("Failed to update number of open transactions");
-        checkRetryable(e, "countOpenTxns()");
-      } finally {
-        close(rs, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      countOpenTxns();
-    }
-  }
-
-  /**
-   * Add min history level entry for each generated txn record
-   * @param dbConn Connection
-   * @param txnIds new transaction ids
-   * @deprecated Remove this method when min_history_level table is dropped
-   * @throws SQLException ex
-   */
-  @Deprecated
-  private void addTxnToMinHistoryLevel(Connection dbConn, List<Long> txnIds, long minOpenTxnId) throws SQLException {
-    if (!useMinHistoryLevel) {
-      return;
-    }
-    // Need to register minimum open txnid for current transactions into MIN_HISTORY table.
-    try (Statement stmt = dbConn.createStatement()) {
-      List<String> rows = txnIds.stream().map(txnId -> txnId + ", " + minOpenTxnId).collect(Collectors.toList());
-
-      // Insert transaction entries into MIN_HISTORY_LEVEL.
-      List<String> inserts =
-          sqlGenerator.createInsertValuesStmt("\"MIN_HISTORY_LEVEL\" (\"MHL_TXNID\", \"MHL_MIN_OPEN_TXNID\")", rows);
-      for (String insert : inserts) {
-        LOG.debug("Going to execute insert <{}>", insert);
-        stmt.execute(insert);
-      }
-      LOG.info("Added entries to MIN_HISTORY_LEVEL for current txns: ({}) with min_open_txn: {}", txnIds, minOpenTxnId);
-    } catch (SQLException e) {
-      if (dbProduct.isTableNotExistsError(e)) {
-        // If the table does not exists anymore, we disable the flag and start to work the new way
-        // This enables to switch to the new functionality without a restart
-        useMinHistoryLevel = false;
-      } else {
-        throw e;
-      }
+    int openTxns = jdbcResource.execute(new CountOpenTxnsHandler());
+    if (openTxns > -1) {
+      numOpenTxns.set(openTxns);
     }
   }
 
   @Override
   @RetrySemantics.SafeToRetry

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425298034


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -712,6 +782,7 @@ Set<CompactionInfo> findPotentialCompactions(int abortedThreshold, long abortedT
    * WriteSet tracking is used to ensure proper transaction isolation.  This method deletes the
    * transaction metadata once it becomes unnecessary.
    */
+  @Transactional(POOL_TX)

Review Comment:
   ok, done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425269448


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,260 +183,171 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
-  
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
 
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
+  
+  public static class ConfVars {    
 
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
-  private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
+    // Whether to use min_history_level table or not.
+    // At startup we read it from the config, but set it to false if min_history_level does nto exists.
+    private boolean useMinHistoryLevel;
+    private boolean useMinHistoryWriteId;
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
+    public boolean useMinHistoryLevel() {
+      return useMinHistoryLevel;
+    }
 
-  private static final String MANUAL_RETRY = "ManualRetry";
+    public void setUseMinHistoryLevel(boolean useMinHistoryLevel) {
+      this.useMinHistoryLevel = useMinHistoryLevel;
+    }
 
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
+    public boolean useMinHistoryWriteId() {
+      return useMinHistoryWriteId;
+    }
 
+    public void setUseMinHistoryWriteId(boolean useMinHistoryWriteId) {
+      this.useMinHistoryWriteId = useMinHistoryWriteId;
+    }
 
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
+    public void init(BiPredicate<String, Boolean> tableCheck, Configuration conf){
+      useMinHistoryWriteId = tableCheck.test("MIN_HISTORY_WRITE_ID",
+          MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
+      useMinHistoryLevel = tableCheck.test("MIN_HISTORY_LEVEL",
+          MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_LEVEL));
+      
+    }
+  }
+  
+  private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
+  public static final TxnHandler.ConfVars ConfVars = new ConfVars();
 
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockManager txnLockManager;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  @Override
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf,
+                  MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.HIVE_COMPACTOR_CONNECTION_POOLING_MAX_CONNECTIONS));
+            }
+          }
+          if (dbProduct == null) {
+            try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPool)) {
+              determineDatabaseProduct(dbConn);
+            } catch (SQLException e) {
+              LOG.error("Unable to determine database product", e);
+              throw new RuntimeException(e);
+            }
+          }
+          if (sqlGenerator == null) {
+            sqlGenerator = new SQLGenerator(dbProduct, conf);
+          }
+          initialized = true;
         }
       }
+    }
 
-      if (sqlGenerator == null) {
-        sqlGenerator = new SQLGenerator(dbProduct, conf);
-      }
-      
-      if (jdbcResource == null) {
-        jdbcResource = new MultiDataSourceJdbcResource(dbProduct);
-        jdbcResource.registerDataSource(POOL_TX, connPool);
-        jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
-      }      
+    if (jdbcResource == null) {
+      jdbcResource = new MultiDataSourceJdbcResource(dbProduct, conf, sqlGenerator);
+      jdbcResource.registerDataSource(POOL_TX, connPool);
+      jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
+      jdbcResource.registerDataSource(POOL_COMPACTOR, connPoolCompactor);
     }
 
+    mutexAPI = new TxnStoreMutex(sqlGenerator, jdbcResource);
+
     numOpenTxns = Metrics.getOrCreateGauge(MetricsConstants.NUM_OPEN_TXNS);
 
-    timeout = MetastoreConf.getTimeVar(conf, ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    replicationTxnTimeout = MetastoreConf.getTimeVar(conf, ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    retryInterval = MetastoreConf.getTimeVar(conf, ConfVars.HMS_HANDLER_INTERVAL,
-        TimeUnit.MILLISECONDS);
-    retryLimit = MetastoreConf.getIntVar(conf, ConfVars.HMS_HANDLER_ATTEMPTS);
-    deadlockRetryInterval = retryInterval / 10;
-    maxOpenTxns = MetastoreConf.getIntVar(conf, ConfVars.MAX_OPEN_TXNS);
-    maxBatchSize = MetastoreConf.getIntVar(conf, ConfVars.JDBC_MAX_BATCH_SIZE);
+    timeout = MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
+    replicationTxnTimeout = MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
+    maxOpenTxns = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.MAX_OPEN_TXNS);
+    openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
 
-    openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    
     try {
-      useMinHistoryWriteId = checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
-      
-      // override the config if table does not exists anymore
-      // this helps to roll out his feature when multiple HMS is accessing the same backend DB
-      useMinHistoryLevel = checkIfTableIsUsable("MIN_HISTORY_LEVEL",
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_LEVEL));
-    } catch (MetaException e) {
+      TxnHandler.ConfVars.init(this::checkIfTableIsUsable, conf);

Review Comment:
   You are right, and it's enough to init it only once. Moved



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,260 +183,171 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
-  
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
 
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
+  
+  public static class ConfVars {    

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #4566: HIVE-27481: TxnHandler cleanup

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1677602971

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![C](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C-16px.png 'C')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [9 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [121 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1768485080

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [15 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [153 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1407546896


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -193,10 +193,17 @@ public boolean isDeadlock(SQLException e) {
 
   /**
    * Is the given exception a table not found exception
-   * @param e Exception
+   * @param ex Exception
    * @return
    */
-  public boolean isTableNotExistsError(SQLException e) {
+  public boolean isTableNotExistsError(Throwable ex) {
+    while (ex != null && !(ex instanceof SQLException) ) {
+      ex = ex.getCause();
+    }
+    if (ex == null) {
+      throw new IllegalArgumentException("No SQLException found in the exception chain!");
+    }
+    SQLException e = (SQLException) ex;    

Review Comment:
   There is no need for check because the preceding code will either result in an `IllegalArgumentException`, or the ex variable will contain an `SQLException` instance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1410641848


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class StackThreadLocal<T> {
+
+  private final ThreadLocal<ContextNode<T>> threadLocal = new ThreadLocal<>();

Review Comment:
   Good Idea, kept `StackThreadLocal`, because it wraps the handling of the stack (and adds a check), but replaced my custom `ContextNode` implementation with a Stack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "zratkai (via GitHub)" <gi...@apache.org>.
zratkai commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1406032016


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -546,41 +553,48 @@ public boolean supportsGetGeneratedKeys() throws MetaException {
     }
   }
 
-  public boolean isDuplicateKeyError(SQLException ex) {
+  public boolean isDuplicateKeyError(Throwable ex) {
+    while (ex != null && !(ex instanceof SQLException) ) {
+      ex = ex.getCause();
+    }
+    if (ex == null) {
+      throw new IllegalArgumentException("No SQLException found in the exception chain!");

Review Comment:
   This part is the same in isTableNotExistsError. Can you move it to a separate method?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418812037


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -546,61 +518,28 @@ public long findMinOpenTxnIdForCleaner() throws MetaException {
   @RetrySemantics.Idempotent
   @Deprecated
   public long findMinTxnIdSeenOpen() {
-    if (!useMinHistoryLevel || useMinHistoryWriteId) {
+    if (!TxnHandlingFeatures.useMinHistoryLevel() || TxnHandlingFeatures.useMinHistoryWriteId()) {

Review Comment:
   ````
   TxnHandler.ConfVars
       .useMinHistoryLevel()
       .useMinHistoryWriteId()
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419073617


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -5785,261 +1068,22 @@ private void timeOutLocks() {
    */
   @RetrySemantics.Idempotent
   public void performTimeOuts() {
-    jdbcResource.bindDataSource(POOL_TX);
-    try (TransactionContext context = jdbcResource.getTransactionManager().getTransaction(PROPAGATION_REQUIRED)) {
-      //We currently commit after selecting the TXNS to abort.  So whether SERIALIZABLE
-      //READ_COMMITTED, the effect is the same.  We could use FOR UPDATE on Select from TXNS
-      //and do the whole performTimeOuts() in a single huge transaction, but the only benefit
-      //would be to make sure someone cannot heartbeat one of these txns at the same time.
-      //The attempt to heartbeat would block and fail immediately after it's unblocked.
-      //With current (RC + multiple txns) implementation it is possible for someone to send
-      //heartbeat at the very end of the expiry interval, and just after the Select from TXNS
-      //is made, in which case heartbeat will succeed but txn will still be Aborted.
-      //Solving this corner case is not worth the perf penalty.  The client should heartbeat in a
-      //timely way.
-      timeOutLocks();
-      while (true) {
-        String s = " \"TXN_ID\" FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN +
-            " AND (" +
-            "\"TXN_TYPE\" != " + TxnType.REPL_CREATED.getValue() +
-            " AND \"TXN_LAST_HEARTBEAT\" <  " + getEpochFn(dbProduct) + "-" + timeout +
-            " OR " +
-            " \"TXN_TYPE\" = " + TxnType.REPL_CREATED.getValue() +
-            " AND \"TXN_LAST_HEARTBEAT\" <  " + getEpochFn(dbProduct) + "-" + replicationTxnTimeout +
-            ")";
-        //safety valve for extreme cases
-        s = sqlGenerator.addLimitClause(10 * TIMED_OUT_TXN_ABORT_BATCH_SIZE, s);
-
-        LOG.debug("Going to execute query <{}>", s);
-        List<List<Long>> timedOutTxns = jdbcResource.getJdbcTemplate().query(s, rs -> {
-          List<List<Long>> txnbatch = new ArrayList<>();
-          List<Long> currentBatch = new ArrayList<>(TIMED_OUT_TXN_ABORT_BATCH_SIZE);
-          while (rs.next()) {
-            currentBatch.add(rs.getLong(1));
-            if (currentBatch.size() == TIMED_OUT_TXN_ABORT_BATCH_SIZE) {
-              txnbatch.add(currentBatch);
-              currentBatch = new ArrayList<>(TIMED_OUT_TXN_ABORT_BATCH_SIZE);
-            }
-          }
-          if (currentBatch.size() > 0) {
-            txnbatch.add(currentBatch);
-          }
-          return txnbatch;
-        });
-        //noinspection DataFlowIssue
-        if (timedOutTxns.size() == 0) {
-          jdbcResource.getTransactionManager().commit(context);
-          return;
-        }
-
-        Object savePoint = context.getTransactionStatus().createSavepoint();
-
-        int numTxnsAborted = 0;
-        for (List<Long> batchToAbort : timedOutTxns) {
-          context.getTransactionStatus().releaseSavepoint(savePoint);
-          savePoint = context.getTransactionStatus().createSavepoint();
-          if (abortTxns(jdbcResource.getConnection(), batchToAbort, true, false, false, TxnErrorMsg.ABORT_TIMEOUT) == batchToAbort.size()) {
-            numTxnsAborted += batchToAbort.size();
-            //todo: add TXNS.COMMENT filed and set it to 'aborted by system due to timeout'
-          } else {
-            //could not abort all txns in this batch - this may happen because in parallel with this
-            //operation there was activity on one of the txns in this batch (commit/abort/heartbeat)
-            //This is not likely but may happen if client experiences long pause between heartbeats or
-            //unusually long/extreme pauses between heartbeat() calls and other logic in checkLock(),
-            //lock(), etc.
-            context.getTransactionStatus().rollbackToSavepoint(savePoint);
-          }
-        }
-        LOG.info("Aborted {} transaction(s) due to timeout", numTxnsAborted);
-        if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-          Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_TIMED_OUT_TXNS).inc(numTxnsAborted);
-        }
-      }
-    } catch (MetaException | SQLException e) {
-      LOG.warn("Aborting timed out transactions failed due to " + e.getMessage(), e);
-    } finally {
-      jdbcResource.unbindDataSource();
-    }
+    new PerformTimeoutsFunction(timeout, replicationTxnTimeout).execute(jdbcResource);
   }
 
   @Override
   @RetrySemantics.ReadOnly
   public void countOpenTxns() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        String s = "SELECT COUNT(*) FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN;
-        LOG.debug("Going to execute query <{}>", s);
-        rs = stmt.executeQuery(s);
-        if (!rs.next()) {
-          LOG.error("Transaction database not properly configured, can't find txn_state from TXNS.");
-        } else {
-          Long numOpen = rs.getLong(1);
-          if (numOpen > Integer.MAX_VALUE) {
-            LOG.error("Open transaction count above {}, can't count that high!", Integer.MAX_VALUE);
-          } else {
-            numOpenTxns.set(numOpen.intValue());
-          }
-        }
-      } catch (SQLException e) {
-        LOG.info("Failed to update number of open transactions");
-        checkRetryable(e, "countOpenTxns()");
-      } finally {
-        close(rs, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      countOpenTxns();
-    }
-  }
-
-  /**
-   * Add min history level entry for each generated txn record
-   * @param dbConn Connection
-   * @param txnIds new transaction ids
-   * @deprecated Remove this method when min_history_level table is dropped
-   * @throws SQLException ex
-   */
-  @Deprecated
-  private void addTxnToMinHistoryLevel(Connection dbConn, List<Long> txnIds, long minOpenTxnId) throws SQLException {
-    if (!useMinHistoryLevel) {
-      return;
-    }
-    // Need to register minimum open txnid for current transactions into MIN_HISTORY table.
-    try (Statement stmt = dbConn.createStatement()) {
-      List<String> rows = txnIds.stream().map(txnId -> txnId + ", " + minOpenTxnId).collect(Collectors.toList());
-
-      // Insert transaction entries into MIN_HISTORY_LEVEL.
-      List<String> inserts =
-          sqlGenerator.createInsertValuesStmt("\"MIN_HISTORY_LEVEL\" (\"MHL_TXNID\", \"MHL_MIN_OPEN_TXNID\")", rows);
-      for (String insert : inserts) {
-        LOG.debug("Going to execute insert <{}>", insert);
-        stmt.execute(insert);
-      }
-      LOG.info("Added entries to MIN_HISTORY_LEVEL for current txns: ({}) with min_open_txn: {}", txnIds, minOpenTxnId);
-    } catch (SQLException e) {
-      if (dbProduct.isTableNotExistsError(e)) {
-        // If the table does not exists anymore, we disable the flag and start to work the new way
-        // This enables to switch to the new functionality without a restart
-        useMinHistoryLevel = false;
-      } else {
-        throw e;
-      }
+    int openTxns = jdbcResource.execute(new CountOpenTxnsHandler());
+    if (openTxns > -1) {
+      numOpenTxns.set(openTxns);
     }
   }
 
   @Override
   @RetrySemantics.SafeToRetry

Review Comment:
   should we keep this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419182862


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/retryhandling/SqlRetryHandler.java:
##########
@@ -123,32 +124,29 @@ public <Result> Result executeWithRetry(SqlRetryCallProperties properties, SqlRe
       if (properties.isLockInternally()) {
         lockInternal();
       }
-      threadLocal.set(new ContextNode<>(threadLocal.get(), new Object()));
+      threadLocal.set(new Object());

Review Comment:
   but why we need to track this if we don't care what is inside. Can't we just have counter + -



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418887194


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/HiveMutex.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionContext;
+import org.apache.hadoop.hive.metastore.txn.retryhandling.SqlRetryHandler;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.dao.DataAccessException;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.sql.SQLException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Semaphore;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnStore.POOL_MUTEX;
+import static org.springframework.transaction.TransactionDefinition.PROPAGATION_REQUIRED;
+
+public class HiveMutex implements TxnStore.MutexAPI {

Review Comment:
   should it be TxnStoreMutex? HiveMutex is too generic.
   also `txn.impl` package looks unrelated, maybe move to the root `txn` package



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418812724


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandlingFeatures.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class TxnHandlingFeatures {

Review Comment:
   You mean make it singleton, rename it to ConfVars,  and nest it in TxnHandler?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418855723


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnLockHandlerImpl.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.hadoop.hive.metastore.api.LockResponse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.ShowLocksRequest;
+import org.apache.hadoop.hive.metastore.api.ShowLocksResponse;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.api.TxnOpenException;
+import org.apache.hadoop.hive.metastore.api.UnlockRequest;
+import org.apache.hadoop.hive.metastore.txn.entities.LockInfo;
+import org.apache.hadoop.hive.metastore.txn.impl.functions.CheckLockFunction;
+import org.apache.hadoop.hive.metastore.txn.impl.functions.EnqueueLockFunction;
+import org.apache.hadoop.hive.metastore.txn.impl.queries.GetLocksByLockId;
+import org.apache.hadoop.hive.metastore.txn.impl.queries.ShowLocksHandler;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.ProgrammaticRollbackException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.sql.Types;
+import java.util.List;
+
+import static org.apache.hadoop.hive.metastore.txn.entities.LockInfo.LOCK_WAITING;
+
+public class TxnLockHandlerImpl implements TxnLockHandler {

Review Comment:
   TxnLockManagerImpl



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419050332


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {
+    int threshold = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD);
+    MetricsInfo metrics = jdbcResource.execute(MetricsInfoHandler.INSTANCE);
+    Set<String> resourceNames = jdbcResource.execute(new TablesWithAbortedTxnsHandler(threshold));
+    metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
+    metrics.setTablesWithXAbortedTxns(resourceNames);    
+    return metrics;
   }
 
-  /**
-   * Heartbeats on the lock table.  This commits, so do not enter it with any state.
-   * Should not be called on a lock that belongs to transaction.
-   */
-  private void heartbeatLock(Connection dbConn, long extLockId)
-    throws NoSuchLockException, SQLException, MetaException {
-    // If the lock id is 0, then there are no locks in this heartbeat
-    if (extLockId == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String updateHeartbeatQuery = "UPDATE \"HIVE_LOCKS\" SET \"HL_LAST_HEARTBEAT\" = " +
-          getEpochFn(dbProduct) + " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute update <{}>", updateHeartbeatQuery);
-      int rc = stmt.executeUpdate(updateHeartbeatQuery);
-      if (rc < 1) {
-        LOG.error("Failure to update last heartbeat for extLockId={}.", extLockId);
-        dbConn.rollback();
-        throw new NoSuchLockException("No such lock: " + JavaUtils.lockIdToString(extLockId));
-      }
-      LOG.debug("Successfully heartbeated for extLockId={}", extLockId);
-      dbConn.commit();
-    }
-  }
 
-  // Heartbeats on the txn table.  This commits, so do not enter it with any state
-  private void heartbeatTxn(Connection dbConn, long txnid)
-    throws NoSuchTxnException, TxnAbortedException, SQLException, MetaException {
-    // If the txnid is 0, then there are no transactions in this heartbeat
-    if (txnid == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String s = "UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-          " WHERE \"TXN_ID\" = " + txnid + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-      LOG.debug("Going to execute update <{}>", s);
-      int rc = stmt.executeUpdate(s);
-      if (rc < 1) {
-        ensureValidTxn(dbConn, txnid, stmt); // This should now throw some useful exception.
-        LOG.error("Can neither heartbeat txn (txnId={}) nor confirm it as invalid.", txnid);
-        dbConn.rollback();
-        throw new NoSuchTxnException("No such txn: " + txnid);
-      }
-      LOG.debug("Successfully heartbeated for txnId={}", txnid);
-      dbConn.commit();
-    }
+  private static void shouldNeverHappen(long txnid) {
+    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
   }
 
   /**
-   * Returns the state of the transaction if it's able to determine it. Some cases where it cannot:
-   * 1. txnid was Aborted/Committed and then GC'd (compacted)
-   * 2. txnid was committed but it didn't modify anything (nothing in COMPLETED_TXN_COMPONENTS)
+   * Retry-by-caller note:
+   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
+   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
    */
-  private TxnStatus findTxnState(long txnid, Statement stmt) throws SQLException, MetaException {
-    String s = "SELECT \"TXN_STATE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnid;
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        s =
-            sqlGenerator.addLimitClause(1, "1 FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_TXNID\" = "
-                + txnid);
-        LOG.debug("Going to execute query <{}>", s);
-        try (ResultSet rs2 = stmt.executeQuery(s)) {
-          if (rs2.next()) {
-            return TxnStatus.COMMITTED;
-          }
-        }
-        // could also check WRITE_SET but that seems overkill
-        return TxnStatus.UNKNOWN;
-      }
-      return TxnStatus.fromString(rs.getString(1));
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void addDynamicPartitions(AddDynamicPartitions rqst) throws NoSuchTxnException,  TxnAbortedException, MetaException {
+    TxnType txnType = jdbcResource.execute(new GetOpenTxnTypeAndLockHandler(sqlGenerator, rqst.getTxnid()));
+    if (txnType == null) {
+      //ensures txn is still there and in expected state
+      new EnsureValidTxnFunction(rqst.getTxnid()).execute(jdbcResource);
+      shouldNeverHappen(rqst.getTxnid());
+    }
+    jdbcResource.execute(new InsertTxnComponentsCommand(rqst));
+    jdbcResource.getJdbcTemplate().update("DELETE FROM \"TXN_COMPONENTS\" " +
+            "WHERE \"TC_TXNID\" = :txnId AND \"TC_DATABASE\" = :dbName AND \"TC_TABLE\" = :tableName AND \"TC_PARTITION\" IS NULL",
+        new MapSqlParameterSource()
+            .addValue("txnId", rqst.getTxnid())
+            .addValue("dbName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getDbname()))
+            .addValue("tableName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getTablename())));
   }
 
   /**
-   * Checks if all the txns in the list are in open state and not read-only.
-   * @param txnIds list of txns to be evaluated for open state/read-only status
-   * @param stmt db statement
-   * @return If all the txns in open state and not read-only, then return true else false
+   * Clean up corresponding records in metastore tables when corresponding object is dropped,
+   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
+   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
+   * operations.
+   * <p>
+   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
    */
-  private boolean isTxnsOpenAndNotReadOnly(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Get the count of txns from the given list that are in open state and not read-only.
-    // If the returned count is same as the input number of txns, then all txns are in open state and not read-only.
-    prefix.append("SELECT COUNT(*) FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN
-        + " AND \"TXN_TYPE\" != " + TxnType.READ_ONLY.getValue() + " AND ");
-
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-
-    long count = 0;
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        if (rs.next()) {
-          count += rs.getLong(1);
-        }
-      }
-    }
-    return count == txnIds.size();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), keepTxnToWriteIdMetaData, null)
+        .execute(jdbcResource);
   }
 
-  /**
-   * Get txns from the list that are either aborted or read-only.
-   * @param txnIds list of txns to be evaluated for aborted state/read-only status
-   * @param stmt db statement
-   */
-  private String getAbortedAndReadOnlyTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are either aborted or read-only.
-    prefix.append("SELECT \"TXN_ID\", \"TXN_STATE\", \"TXN_TYPE\" FROM \"TXNS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          TxnStatus txnState = TxnStatus.fromString(rs.getString(2));
-          TxnType txnType = TxnType.findByValue(rs.getInt(3));
-
-          if (txnState != TxnStatus.OPEN) {
-            txnInfo.append("{").append(txnId).append(",").append(txnState).append("}");
-          } else if (txnType == TxnType.READ_ONLY) {
-            txnInfo.append("{").append(txnId).append(",read-only}");
-          }
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), false, txnId)
+        .execute(jdbcResource);
   }
-
+  
   /**
-   * Get txns from the list that are committed.
-   * @param txnIds list of txns to be evaluated for committed state
-   * @param stmt db statement
+   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
    */
-  private String getCommittedTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are committed.
-    prefix.append("SELECT \"CTC_TXNID\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"CTC_TXNID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          txnInfo.append("{").append(txnId).append(",c}");
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
+      String newCatName, String newDbName, String newTabName, String newPartName)
+      throws MetaException {
+    new OnRenameFunction(oldCatName, oldDbName, oldTabName, oldPartName, newCatName, newDbName, newTabName, newPartName).execute(jdbcResource);
   }
-
   /**
-   * Used to raise an informative error when the caller expected a txn in a particular TxnStatus
-   * but found it in some other status
+   * For testing only, do not use.
    */
-  private static void raiseTxnUnexpectedState(TxnStatus actualStatus, long txnid)
-    throws NoSuchTxnException, TxnAbortedException {
-    switch (actualStatus) {
-      case ABORTED:
-        throw new TxnAbortedException("Transaction " + JavaUtils.txnIdToString(txnid) + " already aborted");
-      case COMMITTED:
-        throw new NoSuchTxnException("Transaction " + JavaUtils.txnIdToString(txnid) + " is already committed.");
-      case UNKNOWN:
-        throw new NoSuchTxnException("No such transaction " + JavaUtils.txnIdToString(txnid));
-      case OPEN:
-        throw new NoSuchTxnException(JavaUtils.txnIdToString(txnid) + " is " + TxnStatus.OPEN);
-      default:
-        throw new IllegalArgumentException("Unknown TxnStatus " + actualStatus);
-    }
+  @VisibleForTesting
+  public int numLocksInLockTable() {
+    return Objects.requireNonNull(
+        jdbcResource.getJdbcTemplate().queryForObject("SELECT COUNT(*) FROM \"HIVE_LOCKS\"", new MapSqlParameterSource(), Integer.TYPE),
+        "This never should be null, it's just to suppress warnings");
   }
+
   /**
-   * Returns the state of the transaction with {@code txnid} or throws if {@code raiseError} is true.
+   * For testing only, do not use.

Review Comment:
   we shouldn't place test stuff in production code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419107740


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +182,141 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+  public static class TxnHandlingFeatures {    

Review Comment:
   please rename to ConfVars. 
   ````
   public static final ConfVars confVars;
   
   confVars = new ConfVars(table->checkIfTableIsUsable(table), conf)
   
   public static final ConfVars {
     private boolean useMinHistoryLevel;
     private boolean useMinHistoryWriteId;
     
     public ConfVars(Predicate<String> tableCheck, Conf conf){
       useMinHistoryLevel = tableCheck.test("MIN_HISTORY_WRITE_ID", 
     MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))
     }
   }
   ````
   
   getConfVars().useMinHistoryWriteId
   getConfVars().useMinHistoryLevel



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419107740


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +182,141 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+  public static class TxnHandlingFeatures {    

Review Comment:
   please rename to ConfVars. 
   ````
   ConfVars.init(table->checkIfTableIsUsable(table), conf)
   
   public static final ConfVars {
     private boolean useMinHistoryLevel;
     private boolean useMinHistoryWriteId;
     
     public init(Predicate<String> tableCheck, Conf conf){
       useMinHistoryLevel = tableCheck.test("MIN_HISTORY_WRITE_ID", 
     MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))
     }
   }
   ````
   
   ConfVars.get().useMinHistoryWriteId
   ConfVars.get().useMinHistoryLevel



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419150687


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import java.util.Stack;
+
+public class StackThreadLocal<T> {

Review Comment:
   maybe org.apache.hadoop.hive.metastore.utils?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425284878


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +431,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table
+    StopWatch generateTransactionWatch = new StopWatch();
+    generateTransactionWatch.start();
 
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        generateTransactionWatch.stop();
-        long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
-        TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-        if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
-          /*
-           * The commit was too slow, we can not allow this to continue (except if it is read only,
-           * since that can not cause dirty reads).
-           * When calculating the snapshot for a given transaction, we look back for possible open transactions
-           * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
-           * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
-           * because there can be other transactions running, that didn't considered this transactionId open,
-           * this could cause dirty reads.
-           */
-          LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
-          deleteInvalidOpenTransactions(dbConn, txnIds);
-          dbConn.commit();
-          /*
-           * We do not throw RetryException directly, to not circumvent the max retry limit
-           */
-          throw new SQLException("OpenTxnTimeOut exceeded", MANUAL_RETRY);
-        }
-        return new OpenTxnsResponse(txnIds);
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "openTxns(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return openTxns(rqst);
-    }
-  }
+    List<Long> txnIds = new OpenTxnsFunction(rqst, openTxnTimeOutMillis, transactionalListeners).execute(jdbcResource);
 
-  private List<Long> openTxns(Connection dbConn, OpenTxnRequest rqst)
-          throws SQLException, MetaException {
-    int numTxns = rqst.getNum_txns();
-    // Make sure the user has not requested an insane amount of txns.
-    int maxTxns = MetastoreConf.getIntVar(conf, ConfVars.TXN_MAX_OPEN_BATCH);
-    if (numTxns > maxTxns) {
-      numTxns = maxTxns;
-    }
-    List<PreparedStatement> insertPreparedStmts = null;
+    LOG.debug("Going to commit");
+    jdbcResource.getTransactionManager().getActiveTransaction().createSavepoint();

Review Comment:
   what is the line number, is that `dbConn.commit();`? if yes, why don't we call `commit` instead of `getActiveTransaction().createSavepoint()`   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1398834938


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -112,104 +51,135 @@
 import org.apache.hadoop.hive.metastore.api.HeartbeatTxnRangeRequest;
 import org.apache.hadoop.hive.metastore.api.HeartbeatTxnRangeResponse;
 import org.apache.hadoop.hive.metastore.api.HiveObjectType;
-import org.apache.hadoop.hive.metastore.api.LockComponent;
 import org.apache.hadoop.hive.metastore.api.LockRequest;
 import org.apache.hadoop.hive.metastore.api.LockResponse;
-import org.apache.hadoop.hive.metastore.api.LockState;
-import org.apache.hadoop.hive.metastore.api.LockType;
 import org.apache.hadoop.hive.metastore.api.Materialization;
 import org.apache.hadoop.hive.metastore.api.MaxAllocatedTableWriteIdRequest;
 import org.apache.hadoop.hive.metastore.api.MaxAllocatedTableWriteIdResponse;
 import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchCompactionException;
 import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
 import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
 import org.apache.hadoop.hive.metastore.api.OpenTxnRequest;
 import org.apache.hadoop.hive.metastore.api.OpenTxnsResponse;
 import org.apache.hadoop.hive.metastore.api.Partition;
-import org.apache.hadoop.hive.metastore.api.ReplLastIdInfo;
 import org.apache.hadoop.hive.metastore.api.ReplTblWriteIdStateRequest;
 import org.apache.hadoop.hive.metastore.api.SeedTableWriteIdsRequest;
 import org.apache.hadoop.hive.metastore.api.SeedTxnIdRequest;
 import org.apache.hadoop.hive.metastore.api.ShowCompactRequest;
 import org.apache.hadoop.hive.metastore.api.ShowCompactResponse;
-import org.apache.hadoop.hive.metastore.api.ShowCompactResponseElement;
 import org.apache.hadoop.hive.metastore.api.ShowLocksRequest;
 import org.apache.hadoop.hive.metastore.api.ShowLocksResponse;
-import org.apache.hadoop.hive.metastore.api.ShowLocksResponseElement;
 import org.apache.hadoop.hive.metastore.api.Table;
-import org.apache.hadoop.hive.metastore.api.TableValidWriteIds;
 import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
 import org.apache.hadoop.hive.metastore.api.TxnOpenException;
-import org.apache.hadoop.hive.metastore.api.TxnToWriteId;
 import org.apache.hadoop.hive.metastore.api.TxnType;
 import org.apache.hadoop.hive.metastore.api.UnlockRequest;
 import org.apache.hadoop.hive.metastore.api.UpdateTransactionalStatsRequest;
-import org.apache.hadoop.hive.metastore.api.WriteEventInfo;
 import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
 import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
 import org.apache.hadoop.hive.metastore.datasource.DataSourceProvider;
 import org.apache.hadoop.hive.metastore.datasource.DataSourceProviderFactory;
 import org.apache.hadoop.hive.metastore.events.AbortTxnEvent;
-import org.apache.hadoop.hive.metastore.events.AllocWriteIdEvent;
-import org.apache.hadoop.hive.metastore.events.CommitTxnEvent;
-import org.apache.hadoop.hive.metastore.events.ListenerEvent;
-import org.apache.hadoop.hive.metastore.events.OpenTxnEvent;
 import org.apache.hadoop.hive.metastore.events.AcidWriteEvent;
+import org.apache.hadoop.hive.metastore.events.ListenerEvent;
 import org.apache.hadoop.hive.metastore.messaging.EventMessage;
 import org.apache.hadoop.hive.metastore.metrics.Metrics;
 import org.apache.hadoop.hive.metastore.metrics.MetricsConstants;
 import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
-import org.apache.hadoop.hive.metastore.txn.impl.InsertCompactionInfoCommand;
-import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionContext;
-import org.apache.hadoop.hive.metastore.txn.retryhandling.SqlRetryCallProperties;
+import org.apache.hadoop.hive.metastore.txn.entities.LockInfo;
+import org.apache.hadoop.hive.metastore.txn.impl.HiveMutex;
+import org.apache.hadoop.hive.metastore.txn.impl.commands.AddWriteIdsToMinHistoryCommand;

Review Comment:
   in general we're against wildcard imports, but this time it's a typical usecase to avoid mass import and have a clean representation of the new class hierarchy:
   ```
   import org.apache.hadoop.hive.metastore.txn.impl.commands.*;
   import org.apache.hadoop.hive.metastore.txn.impl.functions.*;
   import org.apache.hadoop.hive.metastore.txn.impl.queries.*;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1398831631


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ParameterizedBatchCommand.java:
##########
@@ -0,0 +1,61 @@
+/*

Review Comment:
   maybe it's just a matter taste, but I feel that an interface should go into the same package where it's typically implemented by many classes, if that package is typically about to contain those particular classes, so instead of having org.apache.hadoop.hive.metastore.txn.jdbc.ParameterizedBatchCommand and its subclasses in org.apache.hadoop.hive.metastore.txn.impl.commands, all of them can go into a package "org.apache.hadoop.hive.metastore.txn.command"
   
   what do you think?
   
   and the same applies to functions and other stuff introduced in this PR
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1398821824


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ConditionalCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import org.apache.hadoop.hive.metastore.DatabaseProduct;
+
+/**
+ * {@link ParameterizedCommand}. {@link ParameterizedBatchCommand}, and {@link InClauseBatchCommand} implementations can also
+ * implement this interface, marking that the execution is conditial, and the command wants to get notified about 
+ * execution errors. Can be used to implement commands depending on some feature flag(s).  
+ */
+public interface ConditionalCommand {
+
+  /**
+   * Indicates if the command should be executed or not
+   * @param databaseProduct
+   * @return Returns true if the command can be executed, false otherwise.
+   */
+  boolean shouldUse(DatabaseProduct databaseProduct);

Review Comment:
   I know, grammar nit, but this is an interface, should be as obvious as possible, so this is a command, that should run or not, right? in this case "use" as an active verb is misleading, should be "shouldBeUsed" or rather "shouldRun", because this command should be used / this command should run



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1410699493


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +180,114 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;

Review Comment:
   Actually I mode it back from there. The pool initialization must be done only once, but the `Configuration` object is required to do it, so there is no way to put it in a static initializer block. If I move the pool initialization to `CompactionTxnHandler`, I have to copy paste the surrounding double-checked-locking and try with resources blocks, so I thought it is better to initialize all the pools at the same place.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1410639712


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +180,114 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockHandler txnLockHandler;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf, maxPoolSize);

Review Comment:
   Good catch! for compaction pool that value needs to be used. Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425269628


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +400,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table
+    StopWatch generateTransactionWatch = new StopWatch();
+    generateTransactionWatch.start();
 
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        generateTransactionWatch.stop();
-        long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
-        TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-        if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
-          /*
-           * The commit was too slow, we can not allow this to continue (except if it is read only,
-           * since that can not cause dirty reads).
-           * When calculating the snapshot for a given transaction, we look back for possible open transactions
-           * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
-           * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
-           * because there can be other transactions running, that didn't considered this transactionId open,
-           * this could cause dirty reads.
-           */
-          LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
-          deleteInvalidOpenTransactions(dbConn, txnIds);
-          dbConn.commit();
-          /*
-           * We do not throw RetryException directly, to not circumvent the max retry limit
-           */
-          throw new SQLException("OpenTxnTimeOut exceeded", MANUAL_RETRY);
-        }
-        return new OpenTxnsResponse(txnIds);
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "openTxns(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return openTxns(rqst);
-    }
-  }
+    List<Long> txnIds = new OpenTxnsFunction(rqst, openTxnTimeOutMillis, transactionalListeners).execute(jdbcResource);
 
-  private List<Long> openTxns(Connection dbConn, OpenTxnRequest rqst)
-          throws SQLException, MetaException {
-    int numTxns = rqst.getNum_txns();
-    // Make sure the user has not requested an insane amount of txns.
-    int maxTxns = MetastoreConf.getIntVar(conf, ConfVars.TXN_MAX_OPEN_BATCH);
-    if (numTxns > maxTxns) {
-      numTxns = maxTxns;
-    }
-    List<PreparedStatement> insertPreparedStmts = null;
+    LOG.debug("Going to commit");
+    jdbcResource.getTransactionManager().getActiveTransaction().createSavepoint();
+    generateTransactionWatch.stop();
+    long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
     TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-    boolean isReplayedReplTxn = txnType == TxnType.REPL_CREATED;
-    boolean isHiveReplTxn = rqst.isSetReplPolicy() && txnType == TxnType.DEFAULT;
-    try {
-      if (isReplayedReplTxn) {
-        assert rqst.isSetReplPolicy();
-        List<Long> targetTxnIdList = getTargetTxnIdList(rqst.getReplPolicy(), rqst.getReplSrcTxnIds(), dbConn);
-
-        if (!targetTxnIdList.isEmpty()) {
-          if (targetTxnIdList.size() != rqst.getReplSrcTxnIds().size()) {
-            LOG.warn("target txn id number {} is not matching with source txn id number {}",
-                targetTxnIdList, rqst.getReplSrcTxnIds());
-          }
-          LOG.info("Target transactions {} are present for repl policy : {} and Source transaction id : {}",
-              targetTxnIdList.toString(), rqst.getReplPolicy(), rqst.getReplSrcTxnIds().toString());
-          return targetTxnIdList;
-        }
-      }
-
-      long minOpenTxnId = 0;
-      if (useMinHistoryLevel) {
-        minOpenTxnId = getMinOpenTxnIdWaterMark(dbConn);
-      }
-
-      List<Long> txnIds = new ArrayList<>(numTxns);
+    if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
       /*
-       * The getGeneratedKeys are not supported in every dbms, after executing a multi line insert.
-       * But it is supported in every used dbms for single line insert, even if the metadata says otherwise.
-       * If the getGeneratedKeys are not supported first we insert a random batchId in the TXN_META_INFO field,
-       * then the keys are selected beck with that batchid.
+       * The commit was too slow, we can not allow this to continue (except if it is read only,
+       * since that can not cause dirty reads).
+       * When calculating the snapshot for a given transaction, we look back for possible open transactions
+       * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
+       * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
+       * because there can be other transactions running, that didn't considered this transactionId open,
+       * this could cause dirty reads.
        */
-      boolean genKeySupport = dbProduct.supportsGetGeneratedKeys();
-      genKeySupport = genKeySupport || (numTxns == 1);
-
-      String insertQuery = String.format(TXNS_INSERT_QRY, getEpochFn(dbProduct), getEpochFn(dbProduct));
-      LOG.debug("Going to execute insert <{}>", insertQuery);
-      try (PreparedStatement ps = dbConn.prepareStatement(insertQuery, new String[] {"TXN_ID"})) {
-        String state = genKeySupport ? TxnStatus.OPEN.getSqlConst() : TXN_TMP_STATE;
-        if (numTxns == 1) {
-          ps.setString(1, state);
-          ps.setString(2, rqst.getUser());
-          ps.setString(3, rqst.getHostname());
-          ps.setInt(4, txnType.getValue());
-          txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, false));
-        } else {
-          for (int i = 0; i < numTxns; ++i) {
-            ps.setString(1, state);
-            ps.setString(2, rqst.getUser());
-            ps.setString(3, rqst.getHostname());
-            ps.setInt(4, txnType.getValue());
-            ps.addBatch();
-
-            if ((i + 1) % maxBatchSize == 0) {
-              txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, true));
-            }
-          }
-          if (numTxns % maxBatchSize != 0) {
-            txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, true));
-          }
-        }
-      }
-
-      assert txnIds.size() == numTxns;
-
-      addTxnToMinHistoryLevel(dbConn, txnIds, minOpenTxnId);
-
-      if (isReplayedReplTxn) {
-        List<String> rowsRepl = new ArrayList<>(numTxns);
-        List<String> params = Collections.singletonList(rqst.getReplPolicy());
-        List<List<String>> paramsList = new ArrayList<>(numTxns);
-        for (int i = 0; i < numTxns; i++) {
-          rowsRepl.add("?," + rqst.getReplSrcTxnIds().get(i) + "," + txnIds.get(i));
-          paramsList.add(params);
-        }
-
-        insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                "\"REPL_TXN_MAP\" (\"RTM_REPL_POLICY\", \"RTM_SRC_TXN_ID\", \"RTM_TARGET_TXN_ID\")", rowsRepl,
-                paramsList);
-        for (PreparedStatement pst : insertPreparedStmts) {
-          pst.execute();
-        }
-      }
-
-      if (transactionalListeners != null && !isHiveReplTxn) {
-        MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                EventMessage.EventType.OPEN_TXN, new OpenTxnEvent(txnIds, txnType), dbConn, sqlGenerator);
-      }
-      return txnIds;
-    } finally {
-      if (insertPreparedStmts != null) {
-        for (PreparedStatement pst : insertPreparedStmts) {
-          pst.close();
-        }
-      }
-    }
-  }
+      LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
 
-  private List<Long> executeTxnInsertBatchAndExtractGeneratedKeys(Connection dbConn, boolean genKeySupport,
-      PreparedStatement ps, boolean batch) throws SQLException {
-    List<Long> txnIds = new ArrayList<>();
-    if (batch) {
-      ps.executeBatch();
-    } else {
-      // For slight performance advantage we do not use the executeBatch, when we only have one row
-      ps.execute();
-    }
-    if (genKeySupport) {
-      try (ResultSet generatedKeys = ps.getGeneratedKeys()) {
-        while (generatedKeys.next()) {
-          txnIds.add(generatedKeys.getLong(1));
-        }
-      }
-    } else {
-      try (PreparedStatement pstmt =
-          dbConn.prepareStatement("SELECT \"TXN_ID\" FROM \"TXNS\" WHERE \"TXN_STATE\" = ?")) {
-        pstmt.setString(1, TXN_TMP_STATE);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          while (rs.next()) {
-            txnIds.add(rs.getLong(1));
-          }
+      if (!txnIds.isEmpty()) {
+        try {

Review Comment:
   why not to keep it in deleteInvalidOpenTransactions() private method, better readability



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425264318


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,260 +183,171 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
-  
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
 
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
+  
+  public static class ConfVars {    
 
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
-  private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
+    // Whether to use min_history_level table or not.
+    // At startup we read it from the config, but set it to false if min_history_level does nto exists.
+    private boolean useMinHistoryLevel;
+    private boolean useMinHistoryWriteId;
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
+    public boolean useMinHistoryLevel() {
+      return useMinHistoryLevel;
+    }
 
-  private static final String MANUAL_RETRY = "ManualRetry";
+    public void setUseMinHistoryLevel(boolean useMinHistoryLevel) {
+      this.useMinHistoryLevel = useMinHistoryLevel;
+    }
 
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
+    public boolean useMinHistoryWriteId() {
+      return useMinHistoryWriteId;
+    }
 
+    public void setUseMinHistoryWriteId(boolean useMinHistoryWriteId) {
+      this.useMinHistoryWriteId = useMinHistoryWriteId;
+    }
 
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
+    public void init(BiPredicate<String, Boolean> tableCheck, Configuration conf){
+      useMinHistoryWriteId = tableCheck.test("MIN_HISTORY_WRITE_ID",
+          MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
+      useMinHistoryLevel = tableCheck.test("MIN_HISTORY_LEVEL",
+          MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_LEVEL));
+      
+    }
+  }
+  
+  private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
+  public static final TxnHandler.ConfVars ConfVars = new ConfVars();
 
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockManager txnLockManager;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  @Override
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf,
+                  MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.HIVE_COMPACTOR_CONNECTION_POOLING_MAX_CONNECTIONS));
+            }
+          }
+          if (dbProduct == null) {
+            try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPool)) {
+              determineDatabaseProduct(dbConn);
+            } catch (SQLException e) {
+              LOG.error("Unable to determine database product", e);
+              throw new RuntimeException(e);
+            }
+          }
+          if (sqlGenerator == null) {
+            sqlGenerator = new SQLGenerator(dbProduct, conf);
+          }
+          initialized = true;
         }
       }
+    }
 
-      if (sqlGenerator == null) {
-        sqlGenerator = new SQLGenerator(dbProduct, conf);
-      }
-      
-      if (jdbcResource == null) {
-        jdbcResource = new MultiDataSourceJdbcResource(dbProduct);
-        jdbcResource.registerDataSource(POOL_TX, connPool);
-        jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
-      }      
+    if (jdbcResource == null) {
+      jdbcResource = new MultiDataSourceJdbcResource(dbProduct, conf, sqlGenerator);
+      jdbcResource.registerDataSource(POOL_TX, connPool);
+      jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
+      jdbcResource.registerDataSource(POOL_COMPACTOR, connPoolCompactor);
     }
 
+    mutexAPI = new TxnStoreMutex(sqlGenerator, jdbcResource);
+
     numOpenTxns = Metrics.getOrCreateGauge(MetricsConstants.NUM_OPEN_TXNS);
 
-    timeout = MetastoreConf.getTimeVar(conf, ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    replicationTxnTimeout = MetastoreConf.getTimeVar(conf, ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    retryInterval = MetastoreConf.getTimeVar(conf, ConfVars.HMS_HANDLER_INTERVAL,
-        TimeUnit.MILLISECONDS);
-    retryLimit = MetastoreConf.getIntVar(conf, ConfVars.HMS_HANDLER_ATTEMPTS);
-    deadlockRetryInterval = retryInterval / 10;
-    maxOpenTxns = MetastoreConf.getIntVar(conf, ConfVars.MAX_OPEN_TXNS);
-    maxBatchSize = MetastoreConf.getIntVar(conf, ConfVars.JDBC_MAX_BATCH_SIZE);
+    timeout = MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
+    replicationTxnTimeout = MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
+    maxOpenTxns = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.MAX_OPEN_TXNS);
+    openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, MetastoreConf.ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
 
-    openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    
     try {
-      useMinHistoryWriteId = checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
-      
-      // override the config if table does not exists anymore
-      // this helps to roll out his feature when multiple HMS is accessing the same backend DB
-      useMinHistoryLevel = checkIfTableIsUsable("MIN_HISTORY_LEVEL",
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_LEVEL));
-    } catch (MetaException e) {
+      TxnHandler.ConfVars.init(this::checkIfTableIsUsable, conf);

Review Comment:
   should this be under the synchronized?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418830575


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);

Review Comment:
   should this be under `!context.isCompleted` if?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418846446


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);

Review Comment:
   Rollback will throw the following exception in case of trying to rolling back a completed transaction (`AbstractPlatformTransactionManager`): 
   ```
   throw new IllegalTransactionStateException(
   					"Transaction is already completed - do not call commit or rollback more than once per transaction");
   ```
   
   `setRollbackOnly()` is called to initiate rollback without exception or manual transaction handling. In this case it's an invalid code path if the transaction is completed (committed or rolled back) because it is handled by the "framework". By not putting it inside an if the exception will reveal this invalid setup.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418965140


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +406,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table

Review Comment:
   is that a new functionality?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418872414


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/entities/LockInfo.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.entities;
+
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.hadoop.hive.metastore.api.LockState;
+import org.apache.hadoop.hive.metastore.api.LockType;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.ShowLocksResponseElement;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.apache.hadoop.hive.metastore.utils.LockTypeUtil;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class LockInfo {

Review Comment:
   since you are doing the refactor and created entities package, should we move here `OpenTxn`, `OpenTxnList`, 'CompactionInfo', 'CompactionMetricsData', `MetricsInfo`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418915844


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import java.util.Stack;
+
+public class StackThreadLocal<T> {
+
+  private final ThreadLocal<Stack<T>> threadLocal = new ThreadLocal<>();
+
+  public void set(T value) {
+    Stack<T> stack = threadLocal.get();
+    if (stack == null) {
+      stack = new Stack<>();

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnLockHandler.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import org.apache.hadoop.hive.common.classification.RetrySemantics;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.hadoop.hive.metastore.api.LockResponse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.ShowLocksRequest;
+import org.apache.hadoop.hive.metastore.api.ShowLocksResponse;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.api.TxnOpenException;
+import org.apache.hadoop.hive.metastore.api.UnlockRequest;
+import org.apache.hadoop.hive.metastore.txn.retryhandling.SqlRetry;
+import org.springframework.transaction.annotation.Transactional;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnStore.POOL_TX;
+
+public interface TxnLockHandler {

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -149,15 +164,23 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
           LOG.debug("Successfull method invocation within retry context: {}", callerId);
           return result;
         } catch (IllegalAccessException | InvocationTargetException | UndeclaredThrowableException e) {
-          if (e.getCause() instanceof MetaException) {
-            throw (MetaException) e.getCause();
+          if (e.getCause() instanceof TException) {
+            throw (TException) e.getCause();
           } else if (e.getCause() instanceof RuntimeException) {
             throw (RuntimeException) e.getCause();
           } else {
             throw new RuntimeException(e);
           }
-        } catch (Throwable e) {
-          throw new RuntimeException(e);
+        } catch (TException | DataAccessException e) {

Review Comment:
   I think this block is specific to this class. Can't recall similar blocks elsewhere.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -712,6 +782,7 @@ Set<CompactionInfo> findPotentialCompactions(int abortedThreshold, long abortedT
    * WriteSet tracking is used to ensure proper transaction isolation.  This method deletes the
    * transaction metadata once it becomes unnecessary.
    */
+  @Transactional(POOL_TX)

Review Comment:
   Doublechecked, and the txn pool was used for this. However I can change it.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java:
##########
@@ -666,4 +627,60 @@ public static Character thriftCompactionType2DbType(CompactionType ct) throws Me
   public static String nvl(String input) {
     return input != null ? " = ? " : " IS NULL ";
   }
+
+  public static String normalizePartitionCase(String s) {

Review Comment:
   `PartitionUtils` is in hive-exec which is not a dependency of metastore-server.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/HiveMutex.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionContext;
+import org.apache.hadoop.hive.metastore.txn.retryhandling.SqlRetryHandler;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.dao.DataAccessException;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.sql.SQLException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Semaphore;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnStore.POOL_MUTEX;
+import static org.springframework.transaction.TransactionDefinition.PROPAGATION_REQUIRED;
+
+public class HiveMutex implements TxnStore.MutexAPI {

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/entities/LockInfo.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.entities;
+
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.hadoop.hive.metastore.api.LockState;
+import org.apache.hadoop.hive.metastore.api.LockType;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.ShowLocksResponseElement;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.apache.hadoop.hive.metastore.utils.LockTypeUtil;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class LockInfo {

Review Comment:
   Yeah good idea, will move them



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/commands/AddWriteIdsToMinHistoryCommand.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.commands;

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/LockHandleImpl.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl;
+
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionContext;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.dao.DataAccessException;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.util.concurrent.Semaphore;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnStore.POOL_MUTEX;
+
+public final class LockHandleImpl implements TxnStore.MutexAPI.LockHandle {

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ConditionalCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import org.apache.hadoop.hive.metastore.DatabaseProduct;
+
+/**
+ * {@link ParameterizedCommand}. {@link ParameterizedBatchCommand}, and {@link InClauseBatchCommand} implementations can also
+ * implement this interface, marking that the execution is conditial, and the command wants to get notified about 
+ * execution errors. Can be used to implement commands depending on some feature flag(s).  
+ */
+public interface ConditionalCommand {
+
+  /**
+   * Indicates if the command should be executed or not
+   * @param databaseProduct
+   * @return Returns true if the command can be executed, false otherwise.
+   */
+  boolean shouldBeUsed(DatabaseProduct databaseProduct);

Review Comment:
   https://github.com/apache/hive/pull/4566/files#diff-99f1624b0fe7acb8f8919dd47ae38747c1abdf98590fe96962cc1bd1bc3fe627R341



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ProgrammaticRollbackException.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import java.lang.reflect.Method;
+
+/**
+ * This exception can be used to trigger rollback in 
+ * {@link org.apache.hadoop.hive.metastore.txn.TransactionalRetryProxy#invoke(Object, Method, Object[])}
+ * for the current transaction, without propagating the exception to the caller. The proxy will catch this exception,
+ * rollback the transaction (if not yet completed already) and return the value supplied in the constructor to te caller.
+ */
+public class ProgrammaticRollbackException extends RuntimeException {  

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/functions/HeartBeatLockFunction.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.functions;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionalFunction;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnUtils.getEpochFn;
+
+/**
+ * Heartbeats on the lock table.  This commits, so do not enter it with any state.
+ * Should not be called on a lock that belongs to transaction.
+ */
+public class HeartBeatLockFunction implements TransactionalFunction<Void> {

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -546,41 +548,42 @@ public boolean supportsGetGeneratedKeys() throws MetaException {
     }
   }
 
-  public boolean isDuplicateKeyError(SQLException ex) {
+  public boolean isDuplicateKeyError(Throwable ex) {

Review Comment:
   done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +406,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table

Review Comment:
   No, this already existed



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -514,112 +365,19 @@ public Configuration getConf() {
   @Override
   @RetrySemantics.ReadOnly
   public GetOpenTxnsInfoResponse getOpenTxnsInfo() throws MetaException {
-    return getOpenTxnsList(true).toOpenTxnsInfoResponse();
+    return jdbcResource.execute(new GetOpenTxnsListHandler(true, openTxnTimeOutMillis)).toOpenTxnsInfoResponse();
   }
 
   @Override
   @RetrySemantics.ReadOnly
   public GetOpenTxnsResponse getOpenTxns() throws MetaException {
-    return getOpenTxnsList(false).toOpenTxnsResponse(Arrays.asList(TxnType.READ_ONLY));
-  }
-
-  private GetOpenTxnsResponse getOpenTxns(Connection dbConn) throws MetaException {
-    return getOpenTxnsList(false, dbConn).toOpenTxnsResponse(Arrays.asList(TxnType.READ_ONLY));
+    return jdbcResource.execute(new GetOpenTxnsListHandler(false, openTxnTimeOutMillis)).toOpenTxnsResponse(Collections.singletonList(TxnType.READ_ONLY));

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -470,28 +320,29 @@ private boolean checkIfTableIsUsable(String tableName, boolean configValue) thro
       // don't check it if disabled
       return false;
     }
-    Connection dbConn = null;
-    boolean tableExists = true;
+    jdbcResource.bindDataSource(POOL_TX);
     try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      try (Statement stmt = dbConn.createStatement()) {
-        // Dummy query to see if table exists
-        try (ResultSet rs = stmt.executeQuery("SELECT 1 FROM \"" + tableName + "\"")) {
-          rs.next();
-        }
-      }
-    } catch (SQLException e) {
+      jdbcResource.getJdbcTemplate().query("SELECT 1 FROM \"" + tableName + "\"", new MapSqlParameterSource(),
+          rs -> {
+            if (rs.next()) {

Review Comment:
   Yeah, I just modified it to use `JdbcTemplate` without analyzing it deeper. The result doesn't matter so I replaced the lambda with `ResultSet::next`



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnLockHandlerImpl.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.hadoop.hive.metastore.api.LockResponse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.ShowLocksRequest;
+import org.apache.hadoop.hive.metastore.api.ShowLocksResponse;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.api.TxnOpenException;
+import org.apache.hadoop.hive.metastore.api.UnlockRequest;
+import org.apache.hadoop.hive.metastore.txn.entities.LockInfo;
+import org.apache.hadoop.hive.metastore.txn.impl.functions.CheckLockFunction;
+import org.apache.hadoop.hive.metastore.txn.impl.functions.EnqueueLockFunction;
+import org.apache.hadoop.hive.metastore.txn.impl.queries.GetLocksByLockId;
+import org.apache.hadoop.hive.metastore.txn.impl.queries.ShowLocksHandler;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.ProgrammaticRollbackException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.sql.Types;
+import java.util.List;
+
+import static org.apache.hadoop.hive.metastore.txn.entities.LockInfo.LOCK_WAITING;
+
+public class TxnLockHandlerImpl implements TxnLockHandler {

Review Comment:
   Done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +181,115 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockHandler txnLockHandler;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf,
+                  MetastoreConf.getIntVar(conf, ConfVars.HIVE_COMPACTOR_CONNECTION_POOLING_MAX_CONNECTIONS));
+            }            
+          }
+          if (dbProduct == null) {
+            try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPool)) {
+              determineDatabaseProduct(dbConn);
+            } catch (SQLException e) {
+              LOG.error("Unable to determine database product", e);
+              throw new RuntimeException(e);
+            }
+          }
+          if (sqlGenerator == null) {
+            sqlGenerator = new SQLGenerator(dbProduct, conf);
+          }
+          initialized = true;
         }
       }
+    }
 
-      if (sqlGenerator == null) {
-        sqlGenerator = new SQLGenerator(dbProduct, conf);
-      }
-      
-      if (jdbcResource == null) {
-        jdbcResource = new MultiDataSourceJdbcResource(dbProduct);
-        jdbcResource.registerDataSource(POOL_TX, connPool);
-        jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
-      }      
+    if (jdbcResource == null) {
+      jdbcResource = new MultiDataSourceJdbcResource(dbProduct, conf, sqlGenerator);
+      jdbcResource.registerDataSource(POOL_TX, connPool);
+      jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
+      jdbcResource.registerDataSource(POOL_COMPACTOR, connPoolCompactor);
     }
+    
+    mutexAPI = new HiveMutex(sqlGenerator, jdbcResource);
 
     numOpenTxns = Metrics.getOrCreateGauge(MetricsConstants.NUM_OPEN_TXNS);
 
     timeout = MetastoreConf.getTimeVar(conf, ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
     replicationTxnTimeout = MetastoreConf.getTimeVar(conf, ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    retryInterval = MetastoreConf.getTimeVar(conf, ConfVars.HMS_HANDLER_INTERVAL,
-        TimeUnit.MILLISECONDS);
-    retryLimit = MetastoreConf.getIntVar(conf, ConfVars.HMS_HANDLER_ATTEMPTS);
-    deadlockRetryInterval = retryInterval / 10;
     maxOpenTxns = MetastoreConf.getIntVar(conf, ConfVars.MAX_OPEN_TXNS);
-    maxBatchSize = MetastoreConf.getIntVar(conf, ConfVars.JDBC_MAX_BATCH_SIZE);
-
     openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
     
     try {
-      useMinHistoryWriteId = checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
-      
-      // override the config if table does not exists anymore
+      TxnHandlingFeatures.setUseMinHistoryWriteId(checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
+              MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))

Review Comment:
   Did the formatting. Moving it would require to pass `jdbcResource` as well, I think it is easier to set up the fields as a part of TxnHandler init.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/retryhandling/SqlRetryHandler.java:
##########
@@ -123,32 +124,29 @@ public <Result> Result executeWithRetry(SqlRetryCallProperties properties, SqlRe
       if (properties.isLockInternally()) {
         lockInternal();
       }
-      threadLocal.set(new ContextNode<>(threadLocal.get(), new Object()));
+      threadLocal.set(new Object());

Review Comment:
   In `SqlRetryHandler` the context doesn't hold any information. It's just a marker if we already have an established context or not. The reason it's not a boolean flag that theoretically there can be any level of nesting which we need to keep track of.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -457,7 +306,8 @@ public void setConf(Configuration conf){
       throw new RuntimeException(e);
     }
 
-    sqlRetryHandler = new SqlRetryHandler(conf, jdbcResource.getDatabaseProduct());    
+    sqlRetryHandler = new SqlRetryHandler(conf, jdbcResource.getDatabaseProduct());
+    txnLockHandler = TransactionalRetryProxy.getProxy(new TxnLockHandlerImpl(jdbcResource), sqlRetryHandler, jdbcResource);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419157502


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);

Review Comment:
   ok, I just saw !context.isCompleted check before rollback here
   ````
             if (context != null && !context.isCompleted()) {
               jdbcResource.getTransactionManager().rollback(context);
             } 
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1840712220

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [228 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422543987


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {
+    int threshold = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD);
+    MetricsInfo metrics = jdbcResource.execute(MetricsInfoHandler.INSTANCE);
+    Set<String> resourceNames = jdbcResource.execute(new TablesWithAbortedTxnsHandler(threshold));
+    metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
+    metrics.setTablesWithXAbortedTxns(resourceNames);    
+    return metrics;
   }
 
-  /**
-   * Heartbeats on the lock table.  This commits, so do not enter it with any state.
-   * Should not be called on a lock that belongs to transaction.
-   */
-  private void heartbeatLock(Connection dbConn, long extLockId)
-    throws NoSuchLockException, SQLException, MetaException {
-    // If the lock id is 0, then there are no locks in this heartbeat
-    if (extLockId == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String updateHeartbeatQuery = "UPDATE \"HIVE_LOCKS\" SET \"HL_LAST_HEARTBEAT\" = " +
-          getEpochFn(dbProduct) + " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute update <{}>", updateHeartbeatQuery);
-      int rc = stmt.executeUpdate(updateHeartbeatQuery);
-      if (rc < 1) {
-        LOG.error("Failure to update last heartbeat for extLockId={}.", extLockId);
-        dbConn.rollback();
-        throw new NoSuchLockException("No such lock: " + JavaUtils.lockIdToString(extLockId));
-      }
-      LOG.debug("Successfully heartbeated for extLockId={}", extLockId);
-      dbConn.commit();
-    }
-  }
 
-  // Heartbeats on the txn table.  This commits, so do not enter it with any state
-  private void heartbeatTxn(Connection dbConn, long txnid)
-    throws NoSuchTxnException, TxnAbortedException, SQLException, MetaException {
-    // If the txnid is 0, then there are no transactions in this heartbeat
-    if (txnid == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String s = "UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-          " WHERE \"TXN_ID\" = " + txnid + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-      LOG.debug("Going to execute update <{}>", s);
-      int rc = stmt.executeUpdate(s);
-      if (rc < 1) {
-        ensureValidTxn(dbConn, txnid, stmt); // This should now throw some useful exception.
-        LOG.error("Can neither heartbeat txn (txnId={}) nor confirm it as invalid.", txnid);
-        dbConn.rollback();
-        throw new NoSuchTxnException("No such txn: " + txnid);
-      }
-      LOG.debug("Successfully heartbeated for txnId={}", txnid);
-      dbConn.commit();
-    }
+  private static void shouldNeverHappen(long txnid) {

Review Comment:
   done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {

Review Comment:
   added to all missing places



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422534487


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -6052,450 +1096,53 @@ protected synchronized static DataSource setupJdbcConnectionPool(Configuration c
         throw new RuntimeException(e);
       }
     } else {
-      String connectionPooler = MetastoreConf.getVar(conf, ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
+      String connectionPooler = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
       if ("none".equals(connectionPooler)) {
         LOG.info("Choosing not to pool JDBC connections");
-        return new NoPoolConnectionPool(conf);
+        return new NoPoolConnectionPool(conf, dbProduct);
       } else {
         throw new RuntimeException("Unknown JDBC connection pooling " + connectionPooler);
       }
     }
   }
 
-  /**
-   * Returns true if {@code ex} should be retried
-   */
-  static boolean isRetryable(Configuration conf, Exception ex) {
-    if(ex instanceof SQLException) {
-      SQLException sqlException = (SQLException)ex;
-      if (MANUAL_RETRY.equalsIgnoreCase(sqlException.getSQLState())) {
-        // Manual retry exception was thrown
-        return true;
-      }
-      if ("08S01".equalsIgnoreCase(sqlException.getSQLState())) {
-        //in MSSQL this means Communication Link Failure
-        return true;
-      }
-      if ("ORA-08176".equalsIgnoreCase(sqlException.getSQLState()) ||
-        sqlException.getMessage().contains("consistent read failure; rollback data not available")) {
-        return true;
-      }
-
-      String regex = MetastoreConf.getVar(conf, ConfVars.TXN_RETRYABLE_SQLEX_REGEX);
-      if (regex != null && !regex.isEmpty()) {
-        String[] patterns = regex.split(",(?=\\S)");
-        String message = getMessage((SQLException)ex);
-        for (String p : patterns) {
-          if (Pattern.matches(p, message)) {
-            return true;
-          }
-        }
-      }
-      //see also https://issues.apache.org/jira/browse/HIVE-9938
-    }
-    return false;
-  }
-
-  private boolean isDuplicateKeyError(SQLException ex) {
-    return dbProduct.isDuplicateKeyError(ex);
-  }
-
-  private static String getMessage(SQLException ex) {
-    return ex.getMessage() + " (SQLState=" + ex.getSQLState() + ", ErrorCode=" + ex.getErrorCode() + ")";
-  }
-  static String quoteString(String input) {
-    return "'" + input + "'";
-  }
-  static String quoteChar(char c) {
-    return "'" + c + "'";
-  }
-
-  /**
-   * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
-   * Select ... For Update to sequence operations properly.  In practice that means when running
-   * with Derby database.  See more notes at class level.
-   */
-  protected void lockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.lock();
-    }
-  }
-  protected void unlockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.unlock();
-    }
-  }
   @Override
   @RetrySemantics.Idempotent
   public MutexAPI getMutexAPI() {

Review Comment:
   removed



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -6052,450 +1096,53 @@ protected synchronized static DataSource setupJdbcConnectionPool(Configuration c
         throw new RuntimeException(e);
       }
     } else {
-      String connectionPooler = MetastoreConf.getVar(conf, ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
+      String connectionPooler = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
       if ("none".equals(connectionPooler)) {
         LOG.info("Choosing not to pool JDBC connections");
-        return new NoPoolConnectionPool(conf);
+        return new NoPoolConnectionPool(conf, dbProduct);
       } else {
         throw new RuntimeException("Unknown JDBC connection pooling " + connectionPooler);
       }
     }
   }
 
-  /**
-   * Returns true if {@code ex} should be retried
-   */
-  static boolean isRetryable(Configuration conf, Exception ex) {
-    if(ex instanceof SQLException) {
-      SQLException sqlException = (SQLException)ex;
-      if (MANUAL_RETRY.equalsIgnoreCase(sqlException.getSQLState())) {
-        // Manual retry exception was thrown
-        return true;
-      }
-      if ("08S01".equalsIgnoreCase(sqlException.getSQLState())) {
-        //in MSSQL this means Communication Link Failure
-        return true;
-      }
-      if ("ORA-08176".equalsIgnoreCase(sqlException.getSQLState()) ||
-        sqlException.getMessage().contains("consistent read failure; rollback data not available")) {
-        return true;
-      }
-
-      String regex = MetastoreConf.getVar(conf, ConfVars.TXN_RETRYABLE_SQLEX_REGEX);
-      if (regex != null && !regex.isEmpty()) {
-        String[] patterns = regex.split(",(?=\\S)");
-        String message = getMessage((SQLException)ex);
-        for (String p : patterns) {
-          if (Pattern.matches(p, message)) {
-            return true;
-          }
-        }
-      }
-      //see also https://issues.apache.org/jira/browse/HIVE-9938
-    }
-    return false;
-  }
-
-  private boolean isDuplicateKeyError(SQLException ex) {
-    return dbProduct.isDuplicateKeyError(ex);
-  }
-
-  private static String getMessage(SQLException ex) {
-    return ex.getMessage() + " (SQLState=" + ex.getSQLState() + ", ErrorCode=" + ex.getErrorCode() + ")";
-  }
-  static String quoteString(String input) {
-    return "'" + input + "'";
-  }
-  static String quoteChar(char c) {
-    return "'" + c + "'";
-  }
-
-  /**
-   * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
-   * Select ... For Update to sequence operations properly.  In practice that means when running
-   * with Derby database.  See more notes at class level.
-   */
-  protected void lockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.lock();
-    }
-  }
-  protected void unlockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.unlock();
-    }
-  }
   @Override
   @RetrySemantics.Idempotent
   public MutexAPI getMutexAPI() {
-    return this;
+    return mutexAPI;
   }
 
   @Override
   public LockHandle acquireLock(String key) throws MetaException {
-    /**
-     * The implementation here is a bit kludgey but done so that code exercised by unit tests
-     * (which run against Derby which has no support for select for update) is as similar to
-     * production code as possible.
-     * In particular, with Derby we always run in a single process with a single metastore and
-     * the absence of For Update is handled via a Semaphore.  The later would strictly speaking
-     * make the SQL statements below unnecessary (for Derby), but then they would not be tested.
-     */
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    boolean needToCloseConn = true;
-    try {
-      try {
-        String sqlStmt = sqlGenerator.addForUpdateClause("SELECT \"MT_COMMENT\", \"MT_KEY2\" FROM \"AUX_TABLE\" WHERE \"MT_KEY1\"=" + quoteString(key));
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPoolMutex);
-        stmt = dbConn.createStatement();
-        LOG.debug("About to execute SQL: {}", sqlStmt);
-        rs = stmt.executeQuery(sqlStmt);
-        if (!rs.next()) {
-          close(rs);
-          try {
-            stmt.executeUpdate("INSERT INTO \"AUX_TABLE\" (\"MT_KEY1\", \"MT_KEY2\") VALUES(" + quoteString(key) + ", 0)");
-            dbConn.commit();
-          } catch (SQLException ex) {
-            if (!isDuplicateKeyError(ex)) {
-              throw new RuntimeException("Unable to lock " + quoteString(key) + " due to: " + getMessage(ex), ex);
-            }
-            //if here, it means a concrurrent acquireLock() inserted the 'key'
-
-            //rollback is done for the benefit of Postgres which throws (SQLState=25P02, ErrorCode=0) if
-            //you attempt any stmt in a txn which had an error.
-            dbConn.rollback();
-          }
-          rs = stmt.executeQuery(sqlStmt);
-          if (!rs.next()) {
-            throw new IllegalStateException("Unable to lock " + quoteString(key) + ".  Expected row in AUX_TABLE is missing.");
-          }
-        }
-        Semaphore derbySemaphore = null;
-        if(dbProduct.isDERBY()) {
-          derbyKey2Lock.putIfAbsent(key, new Semaphore(1));
-          derbySemaphore =  derbyKey2Lock.get(key);
-          derbySemaphore.acquire();
-        }
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("{} locked by {}", quoteString(key), quoteString(TxnHandler.hostname));
-        }
-        needToCloseConn = false;  //The connection is good, we need not close it
-        //OK, so now we have a lock
-        return new LockHandleImpl(dbConn, stmt, rs, key, derbySemaphore);
-      } catch (SQLException ex) {
-        checkRetryable(ex, "acquireLock(" + key + ")");
-        throw new MetaException("Unable to lock " + quoteString(key) + " due to: " + getMessage(ex) + "; " + StringUtils.stringifyException(ex));
-      }
-      catch(InterruptedException ex) {
-        throw new MetaException("Unable to lock " + quoteString(key) + " due to: " + ex.getMessage() + StringUtils.stringifyException(ex));
-      }
-      finally {
-        if (needToCloseConn) {
-          rollbackDBConn(dbConn);
-          close(rs, stmt, dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException ex) {
-      return acquireLock(key);
-    }
+    return mutexAPI.acquireLock(key);
   }
 
   @Override
-  public void acquireLock(String key, LockHandle handle) {
-    //the idea is that this will use LockHandle.dbConn
-    throw new NotImplementedException("acquireLock(String, LockHandle) is not implemented");
+  public void acquireLock(String key, LockHandle handle) throws MetaException {
+    mutexAPI.acquireLock(key, handle);
   }
 
   /**
    * Acquire the global txn lock, used to mutex the openTxn and commitTxn.
    * @param shared either SHARED_READ or EXCLUSIVE
-   * @throws SQLException
    */
-  private void acquireTxnLock(Statement stmt, boolean shared) throws SQLException, MetaException {
+  private void acquireTxnLock(boolean shared) throws MetaException {
     String sqlStmt = sqlGenerator.createTxnLockStatement(shared);
-    stmt.execute(sqlStmt);
-    LOG.debug("TXN lock locked by {} in mode {}", quoteString(TxnHandler.hostname), shared);
-  }
-
-  private static final class LockHandleImpl implements LockHandle {
-    private final Connection dbConn;
-    private final Statement stmt;
-    private final ResultSet rs;
-    private final Semaphore derbySemaphore;
-    private final String key;
-    private final Long lastUpdateTime;
-
-    LockHandleImpl(Connection conn, Statement stmt, ResultSet rs, String key, Semaphore derbySemaphore) {
-      this.dbConn = conn;
-      this.stmt = stmt;
-      this.rs = rs;
-      this.derbySemaphore = derbySemaphore;
-      if(derbySemaphore != null) {
-        //oterwise it may later release permit acquired by someone else
-        assert derbySemaphore.availablePermits() == 0 : "Expected locked Semaphore";
-      }
-      this.key = key;
-      Long lastUpdateTime;
-      try {
-        lastUpdateTime = rs.getLong("MT_KEY2");
-      } catch (SQLException e) {
-        LOG.warn("Couldn't resolve MT_KEY2 for MT_KEY1=" + quoteString(this.key), e);
-        lastUpdateTime = -1L;
-      }
-      this.lastUpdateTime = lastUpdateTime;
-    }
-
-    @Override
-    public void releaseLocks() {
-      rollbackDBConn(dbConn);
-      TxnHandler.close(rs, stmt, dbConn);
-      if(derbySemaphore != null) {
-        derbySemaphore.release();
-      }
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("{} unlocked by {}", quoteString(key), quoteString(TxnHandler.hostname));
-      }
-    }
-
-    @Override
-    public Long getLastUpdateTime() {
-      return lastUpdateTime;
-    }
-
-    @Override
-    public void releaseLocks(Long timestamp) {
-      try {
-        stmt.executeUpdate("UPDATE \"AUX_TABLE\" SET \"MT_KEY2\" = "+ timestamp + " WHERE \"MT_KEY1\"=" + quoteString(key));
-        dbConn.commit();
-      } catch (SQLException ex) {
-        LOG.warn("Unable to update MT_KEY2 value for MT_KEY1=" + key, ex);
-        rollbackDBConn(dbConn);
-      }
-      TxnHandler.close(rs, stmt, dbConn);
-      if(derbySemaphore != null) {
-        derbySemaphore.release();
-      }
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("{} unlocked by {}", quoteString(key), quoteString(TxnHandler.hostname));
-      }
-    }
-
-    @Override
-    public void close() {
-      releaseLocks();
-    }
-
-  }
-
-
-  private static class NoPoolConnectionPool implements DataSource {
-    // Note that this depends on the fact that no-one in this class calls anything but
-    // getConnection.  If you want to use any of the Logger or wrap calls you'll have to
-    // implement them.
-    private final Configuration conf;
-    private Driver driver;
-    private String connString;
-    private String user;
-    private String passwd;
-
-    public NoPoolConnectionPool(Configuration conf) {
-      this.conf = conf;
-    }
-
-    @Override
-    public Connection getConnection() throws SQLException {
-      if (user == null) {
-        user = DataSourceProvider.getMetastoreJdbcUser(conf);
-        passwd = DataSourceProvider.getMetastoreJdbcPasswd(conf);
-      }
-      return getConnection(user, passwd);
-    }
-
-    @Override
-    public Connection getConnection(String username, String password) throws SQLException {
-      // Find the JDBC driver
-      if (driver == null) {
-        String driverName = MetastoreConf.getVar(conf, ConfVars.CONNECTION_DRIVER);
-        if (driverName == null || driverName.equals("")) {
-          String msg = "JDBC driver for transaction db not set in configuration " +
-              "file, need to set " + ConfVars.CONNECTION_DRIVER.getVarname();
-          LOG.error(msg);
-          throw new RuntimeException(msg);
-        }
-        try {
-          LOG.info("Going to load JDBC driver {}", driverName);
-          driver = (Driver) Class.forName(driverName).newInstance();
-        } catch (InstantiationException e) {
-          throw new RuntimeException("Unable to instantiate driver " + driverName + ", " +
-              e.getMessage(), e);
-        } catch (IllegalAccessException e) {
-          throw new RuntimeException(
-              "Unable to access driver " + driverName + ", " + e.getMessage(),
-              e);
-        } catch (ClassNotFoundException e) {
-          throw new RuntimeException("Unable to find driver " + driverName + ", " + e.getMessage(),
-              e);
-        }
-        connString = MetastoreConf.getVar(conf, ConfVars.CONNECT_URL_KEY);
-      }
-
-      try {
-        LOG.info("Connecting to transaction db with connection string {}", connString);
-        Properties connectionProps = new Properties();
-        connectionProps.setProperty("user", username);
-        connectionProps.setProperty("password", password);
-        Connection conn = driver.connect(connString, connectionProps);
-        String prepareStmt = dbProduct != null ? dbProduct.getPrepareTxnStmt() : null;
-        if (prepareStmt != null) {
-          try (Statement stmt = conn.createStatement()) {
-            stmt.execute(prepareStmt);
-          }
-        }
-        conn.setAutoCommit(false);
-        return conn;
-      } catch (SQLException e) {
-        throw new RuntimeException("Unable to connect to transaction manager using " + connString
-            + ", " + e.getMessage(), e);
-      }
-    }
-
-    @Override
-    public PrintWriter getLogWriter() throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public void setLogWriter(PrintWriter out) throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public void setLoginTimeout(int seconds) throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public int getLoginTimeout() throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public java.util.logging.Logger getParentLogger() throws SQLFeatureNotSupportedException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public <T> T unwrap(Class<T> iface) throws SQLException {
-      throw new UnsupportedOperationException();
-    }
-
-    @Override
-    public boolean isWrapperFor(Class<?> iface) throws SQLException {
-      throw new UnsupportedOperationException();
-    }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> {
+        stmt.execute(sqlStmt);
+        return null;
+    });
+    LOG.debug("TXN lock locked by '{}' in mode {}", TxnHandler.hostname, shared);
   }
 
   @Override
   @RetrySemantics.SafeToRetry

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1808178354

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [17 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [165 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425271671


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +400,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table
+    StopWatch generateTransactionWatch = new StopWatch();
+    generateTransactionWatch.start();
 
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        generateTransactionWatch.stop();
-        long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
-        TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-        if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
-          /*
-           * The commit was too slow, we can not allow this to continue (except if it is read only,
-           * since that can not cause dirty reads).
-           * When calculating the snapshot for a given transaction, we look back for possible open transactions
-           * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
-           * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
-           * because there can be other transactions running, that didn't considered this transactionId open,
-           * this could cause dirty reads.
-           */
-          LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
-          deleteInvalidOpenTransactions(dbConn, txnIds);
-          dbConn.commit();
-          /*
-           * We do not throw RetryException directly, to not circumvent the max retry limit
-           */
-          throw new SQLException("OpenTxnTimeOut exceeded", MANUAL_RETRY);
-        }
-        return new OpenTxnsResponse(txnIds);
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "openTxns(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return openTxns(rqst);
-    }
-  }
+    List<Long> txnIds = new OpenTxnsFunction(rqst, openTxnTimeOutMillis, transactionalListeners).execute(jdbcResource);
 
-  private List<Long> openTxns(Connection dbConn, OpenTxnRequest rqst)
-          throws SQLException, MetaException {
-    int numTxns = rqst.getNum_txns();
-    // Make sure the user has not requested an insane amount of txns.
-    int maxTxns = MetastoreConf.getIntVar(conf, ConfVars.TXN_MAX_OPEN_BATCH);
-    if (numTxns > maxTxns) {
-      numTxns = maxTxns;
-    }
-    List<PreparedStatement> insertPreparedStmts = null;
+    LOG.debug("Going to commit");
+    jdbcResource.getTransactionManager().getActiveTransaction().createSavepoint();
+    generateTransactionWatch.stop();
+    long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
     TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-    boolean isReplayedReplTxn = txnType == TxnType.REPL_CREATED;
-    boolean isHiveReplTxn = rqst.isSetReplPolicy() && txnType == TxnType.DEFAULT;
-    try {
-      if (isReplayedReplTxn) {
-        assert rqst.isSetReplPolicy();
-        List<Long> targetTxnIdList = getTargetTxnIdList(rqst.getReplPolicy(), rqst.getReplSrcTxnIds(), dbConn);
-
-        if (!targetTxnIdList.isEmpty()) {
-          if (targetTxnIdList.size() != rqst.getReplSrcTxnIds().size()) {
-            LOG.warn("target txn id number {} is not matching with source txn id number {}",
-                targetTxnIdList, rqst.getReplSrcTxnIds());
-          }
-          LOG.info("Target transactions {} are present for repl policy : {} and Source transaction id : {}",
-              targetTxnIdList.toString(), rqst.getReplPolicy(), rqst.getReplSrcTxnIds().toString());
-          return targetTxnIdList;
-        }
-      }
-
-      long minOpenTxnId = 0;
-      if (useMinHistoryLevel) {
-        minOpenTxnId = getMinOpenTxnIdWaterMark(dbConn);
-      }
-
-      List<Long> txnIds = new ArrayList<>(numTxns);
+    if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
       /*
-       * The getGeneratedKeys are not supported in every dbms, after executing a multi line insert.
-       * But it is supported in every used dbms for single line insert, even if the metadata says otherwise.
-       * If the getGeneratedKeys are not supported first we insert a random batchId in the TXN_META_INFO field,
-       * then the keys are selected beck with that batchid.
+       * The commit was too slow, we can not allow this to continue (except if it is read only,
+       * since that can not cause dirty reads).
+       * When calculating the snapshot for a given transaction, we look back for possible open transactions
+       * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
+       * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
+       * because there can be other transactions running, that didn't considered this transactionId open,
+       * this could cause dirty reads.
        */
-      boolean genKeySupport = dbProduct.supportsGetGeneratedKeys();
-      genKeySupport = genKeySupport || (numTxns == 1);
-
-      String insertQuery = String.format(TXNS_INSERT_QRY, getEpochFn(dbProduct), getEpochFn(dbProduct));
-      LOG.debug("Going to execute insert <{}>", insertQuery);
-      try (PreparedStatement ps = dbConn.prepareStatement(insertQuery, new String[] {"TXN_ID"})) {
-        String state = genKeySupport ? TxnStatus.OPEN.getSqlConst() : TXN_TMP_STATE;
-        if (numTxns == 1) {
-          ps.setString(1, state);
-          ps.setString(2, rqst.getUser());
-          ps.setString(3, rqst.getHostname());
-          ps.setInt(4, txnType.getValue());
-          txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, false));
-        } else {
-          for (int i = 0; i < numTxns; ++i) {
-            ps.setString(1, state);
-            ps.setString(2, rqst.getUser());
-            ps.setString(3, rqst.getHostname());
-            ps.setInt(4, txnType.getValue());
-            ps.addBatch();
-
-            if ((i + 1) % maxBatchSize == 0) {
-              txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, true));
-            }
-          }
-          if (numTxns % maxBatchSize != 0) {
-            txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, true));
-          }
-        }
-      }
-
-      assert txnIds.size() == numTxns;
-
-      addTxnToMinHistoryLevel(dbConn, txnIds, minOpenTxnId);
-
-      if (isReplayedReplTxn) {
-        List<String> rowsRepl = new ArrayList<>(numTxns);
-        List<String> params = Collections.singletonList(rqst.getReplPolicy());
-        List<List<String>> paramsList = new ArrayList<>(numTxns);
-        for (int i = 0; i < numTxns; i++) {
-          rowsRepl.add("?," + rqst.getReplSrcTxnIds().get(i) + "," + txnIds.get(i));
-          paramsList.add(params);
-        }
-
-        insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                "\"REPL_TXN_MAP\" (\"RTM_REPL_POLICY\", \"RTM_SRC_TXN_ID\", \"RTM_TARGET_TXN_ID\")", rowsRepl,
-                paramsList);
-        for (PreparedStatement pst : insertPreparedStmts) {
-          pst.execute();
-        }
-      }
-
-      if (transactionalListeners != null && !isHiveReplTxn) {
-        MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                EventMessage.EventType.OPEN_TXN, new OpenTxnEvent(txnIds, txnType), dbConn, sqlGenerator);
-      }
-      return txnIds;
-    } finally {
-      if (insertPreparedStmts != null) {
-        for (PreparedStatement pst : insertPreparedStmts) {
-          pst.close();
-        }
-      }
-    }
-  }
+      LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
 
-  private List<Long> executeTxnInsertBatchAndExtractGeneratedKeys(Connection dbConn, boolean genKeySupport,
-      PreparedStatement ps, boolean batch) throws SQLException {
-    List<Long> txnIds = new ArrayList<>();
-    if (batch) {
-      ps.executeBatch();
-    } else {
-      // For slight performance advantage we do not use the executeBatch, when we only have one row
-      ps.execute();
-    }
-    if (genKeySupport) {
-      try (ResultSet generatedKeys = ps.getGeneratedKeys()) {
-        while (generatedKeys.next()) {
-          txnIds.add(generatedKeys.getLong(1));
-        }
-      }
-    } else {
-      try (PreparedStatement pstmt =
-          dbConn.prepareStatement("SELECT \"TXN_ID\" FROM \"TXNS\" WHERE \"TXN_STATE\" = ?")) {
-        pstmt.setString(1, TXN_TMP_STATE);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          while (rs.next()) {
-            txnIds.add(rs.getLong(1));
-          }
+      if (!txnIds.isEmpty()) {
+        try {
+          sqlRetryHandler.executeWithRetry(new SqlRetryCallProperties().withCallerId("deleteInvalidOpenTransactions"),
+              () -> {
+                jdbcResource.execute(new DeleteInvalidOpenTxnsCommand(txnIds));
+                LOG.info("Removed transactions: ({}) from TXNS", txnIds);
+                jdbcResource.execute(new RemoveTxnsFromMinHistoryLevelCommand(txnIds));
+                return null;
+              });

Review Comment:
   do we commit the txn here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425298034


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -712,6 +782,7 @@ Set<CompactionInfo> findPotentialCompactions(int abortedThreshold, long abortedT
    * WriteSet tracking is used to ensure proper transaction isolation.  This method deletes the
    * transaction metadata once it becomes unnecessary.
    */
+  @Transactional(POOL_TX)

Review Comment:
   ok, done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1407545337


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -546,41 +553,48 @@ public boolean supportsGetGeneratedKeys() throws MetaException {
     }
   }
 
-  public boolean isDuplicateKeyError(SQLException ex) {
+  public boolean isDuplicateKeyError(Throwable ex) {
+    while (ex != null && !(ex instanceof SQLException) ) {
+      ex = ex.getCause();
+    }
+    if (ex == null) {
+      throw new IllegalArgumentException("No SQLException found in the exception chain!");

Review Comment:
   Created a method in TxnUtils.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1408962234


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java:
##########
@@ -666,4 +627,59 @@ public static Character thriftCompactionType2DbType(CompactionType ct) throws Me
   public static String nvl(String input) {
     return input != null ? " = ? " : " IS NULL ";
   }
+
+  public static String normalizePartitionCase(String s) {
+    if (s == null) {
+      return null;
+    }
+    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
+    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
+  }
+
+  public static long generateTemporaryId() {
+    return -1 * ThreadLocalRandom.current().nextLong();
+  }
+
+  public static boolean isValidTxn(long txnId) {
+    return txnId != 0;
+  }
+
+  /**
+   * Used to raise an informative error when the caller expected a txn in a particular TxnStatus
+   * but found it in some other status
+   */
+  public static void raiseTxnUnexpectedState(TxnStatus actualStatus, long txnid)
+      throws NoSuchTxnException, TxnAbortedException {
+    switch (actualStatus) {
+      case ABORTED:
+        throw new TxnAbortedException("Transaction " + JavaUtils.txnIdToString(txnid) + " already aborted");
+      case COMMITTED:
+        throw new NoSuchTxnException("Transaction " + JavaUtils.txnIdToString(txnid) + " is already committed.");
+      case UNKNOWN:
+        throw new NoSuchTxnException("No such transaction " + JavaUtils.txnIdToString(txnid));
+      case OPEN:
+        throw new NoSuchTxnException(JavaUtils.txnIdToString(txnid) + " is " + TxnStatus.OPEN);
+      default:
+        throw new IllegalArgumentException("Unknown TxnStatus " + actualStatus);
+    }
+  }
+
+  /**
+   * Checks is the passed exception, or any of the root (cause) exceptions are an instance of {@link SQLException}.
+   * Returns with the found {@link SQLException} or throws an {@link IllegalArgumentException} if no {@link SQLException}
+   * found in the chain.
+   * @param ex The exception to check
+   * @return Returns wit the {@link SQLException} found in the exception chain. 

Review Comment:
   typo: wit



##########
ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java:
##########
@@ -107,6 +107,7 @@
 import static junit.framework.Assert.fail;
 import static org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.TABLE_IS_TRANSACTIONAL;
 import static org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.TABLE_TRANSACTIONAL_PROPERTIES;
+import static org.apache.hadoop.hive.metastore.txn.entities.LockInfo.LOCK_WAITING;

Review Comment:
   The code which needs this import is removed by this patch hence this is an unused import.



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class StackThreadLocal<T> {
+
+  private final ThreadLocal<ContextNode<T>> threadLocal = new ThreadLocal<>();

Review Comment:
   How about using `ArrayDeque` or any of the available collection implementations which represent a stack?



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +180,114 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;

Review Comment:
   Should `connPoolCompactor` be moved to `CompactionTxnHandler`?



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +180,114 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockHandler txnLockHandler;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf, maxPoolSize);

Review Comment:
   The original code used HIVE_COMPACTOR_CONNECTION_POOLING_MAX_CONNECTIONS here. Should we keep it? If the answer is not I think it can be removed from MetaStoreConf too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1850749613

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![C](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C-16px.png 'C')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [212 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418694417


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandlingFeatures.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class TxnHandlingFeatures {

Review Comment:
   should it be a singleton instead of class with static fields? + would be better to have it nested in TxnHandler .ConfVars



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418812724


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandlingFeatures.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class TxnHandlingFeatures {

Review Comment:
   You mean make it singleton, rename it to ConfVars,  and nest it in TxnHandler? The problem with Nesting that TxnHandler is currently package private, and the classes under txn.impl package would no longer access the class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418887194


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/HiveMutex.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionContext;
+import org.apache.hadoop.hive.metastore.txn.retryhandling.SqlRetryHandler;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.dao.DataAccessException;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.sql.SQLException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Semaphore;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnStore.POOL_MUTEX;
+import static org.springframework.transaction.TransactionDefinition.PROPAGATION_REQUIRED;
+
+public class HiveMutex implements TxnStore.MutexAPI {

Review Comment:
   should it be TxnStoreMutex? HiveMutex is too generic.
   also `txn.impl` package looks unrelated, maybe move to `txn`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418916474


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ConditionalCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import org.apache.hadoop.hive.metastore.DatabaseProduct;
+
+/**
+ * {@link ParameterizedCommand}. {@link ParameterizedBatchCommand}, and {@link InClauseBatchCommand} implementations can also
+ * implement this interface, marking that the execution is conditial, and the command wants to get notified about 
+ * execution errors. Can be used to implement commands depending on some feature flag(s).  
+ */
+public interface ConditionalCommand {
+
+  /**
+   * Indicates if the command should be executed or not
+   * @param databaseProduct
+   * @return Returns true if the command can be executed, false otherwise.
+   */
+  boolean shouldBeUsed(DatabaseProduct databaseProduct);

Review Comment:
   where is that used?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418962888


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -470,28 +320,29 @@ private boolean checkIfTableIsUsable(String tableName, boolean configValue) thro
       // don't check it if disabled
       return false;
     }
-    Connection dbConn = null;
-    boolean tableExists = true;
+    jdbcResource.bindDataSource(POOL_TX);
     try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      try (Statement stmt = dbConn.createStatement()) {
-        // Dummy query to see if table exists
-        try (ResultSet rs = stmt.executeQuery("SELECT 1 FROM \"" + tableName + "\"")) {
-          rs.next();
-        }
-      }
-    } catch (SQLException e) {
+      jdbcResource.getJdbcTemplate().query("SELECT 1 FROM \"" + tableName + "\"", new MapSqlParameterSource(),
+          rs -> {
+            if (rs.next()) {

Review Comment:
   do we care about the result? i think if table doesn't exist, exception would be thrown before that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419068628


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -6052,450 +1096,53 @@ protected synchronized static DataSource setupJdbcConnectionPool(Configuration c
         throw new RuntimeException(e);
       }
     } else {
-      String connectionPooler = MetastoreConf.getVar(conf, ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
+      String connectionPooler = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
       if ("none".equals(connectionPooler)) {
         LOG.info("Choosing not to pool JDBC connections");
-        return new NoPoolConnectionPool(conf);
+        return new NoPoolConnectionPool(conf, dbProduct);
       } else {
         throw new RuntimeException("Unknown JDBC connection pooling " + connectionPooler);
       }
     }
   }
 
-  /**
-   * Returns true if {@code ex} should be retried
-   */
-  static boolean isRetryable(Configuration conf, Exception ex) {
-    if(ex instanceof SQLException) {
-      SQLException sqlException = (SQLException)ex;
-      if (MANUAL_RETRY.equalsIgnoreCase(sqlException.getSQLState())) {
-        // Manual retry exception was thrown
-        return true;
-      }
-      if ("08S01".equalsIgnoreCase(sqlException.getSQLState())) {
-        //in MSSQL this means Communication Link Failure
-        return true;
-      }
-      if ("ORA-08176".equalsIgnoreCase(sqlException.getSQLState()) ||
-        sqlException.getMessage().contains("consistent read failure; rollback data not available")) {
-        return true;
-      }
-
-      String regex = MetastoreConf.getVar(conf, ConfVars.TXN_RETRYABLE_SQLEX_REGEX);
-      if (regex != null && !regex.isEmpty()) {
-        String[] patterns = regex.split(",(?=\\S)");
-        String message = getMessage((SQLException)ex);
-        for (String p : patterns) {
-          if (Pattern.matches(p, message)) {
-            return true;
-          }
-        }
-      }
-      //see also https://issues.apache.org/jira/browse/HIVE-9938
-    }
-    return false;
-  }
-
-  private boolean isDuplicateKeyError(SQLException ex) {
-    return dbProduct.isDuplicateKeyError(ex);
-  }
-
-  private static String getMessage(SQLException ex) {
-    return ex.getMessage() + " (SQLState=" + ex.getSQLState() + ", ErrorCode=" + ex.getErrorCode() + ")";
-  }
-  static String quoteString(String input) {
-    return "'" + input + "'";
-  }
-  static String quoteChar(char c) {
-    return "'" + c + "'";
-  }
-
-  /**
-   * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
-   * Select ... For Update to sequence operations properly.  In practice that means when running
-   * with Derby database.  See more notes at class level.
-   */
-  protected void lockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.lock();
-    }
-  }
-  protected void unlockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.unlock();
-    }
-  }
   @Override
   @RetrySemantics.Idempotent

Review Comment:
   should we keep this, as it's also present in interface



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419085985


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),

Review Comment:
   why don't we create `lock(rqst)` method in txnLockManager?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419180075


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +181,115 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockHandler txnLockHandler;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf,
+                  MetastoreConf.getIntVar(conf, ConfVars.HIVE_COMPACTOR_CONNECTION_POOLING_MAX_CONNECTIONS));
+            }            
+          }
+          if (dbProduct == null) {
+            try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPool)) {
+              determineDatabaseProduct(dbConn);
+            } catch (SQLException e) {
+              LOG.error("Unable to determine database product", e);
+              throw new RuntimeException(e);
+            }
+          }
+          if (sqlGenerator == null) {
+            sqlGenerator = new SQLGenerator(dbProduct, conf);
+          }
+          initialized = true;
         }
       }
+    }
 
-      if (sqlGenerator == null) {
-        sqlGenerator = new SQLGenerator(dbProduct, conf);
-      }
-      
-      if (jdbcResource == null) {
-        jdbcResource = new MultiDataSourceJdbcResource(dbProduct);
-        jdbcResource.registerDataSource(POOL_TX, connPool);
-        jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
-      }      
+    if (jdbcResource == null) {
+      jdbcResource = new MultiDataSourceJdbcResource(dbProduct, conf, sqlGenerator);
+      jdbcResource.registerDataSource(POOL_TX, connPool);
+      jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
+      jdbcResource.registerDataSource(POOL_COMPACTOR, connPoolCompactor);
     }
+    
+    mutexAPI = new HiveMutex(sqlGenerator, jdbcResource);
 
     numOpenTxns = Metrics.getOrCreateGauge(MetricsConstants.NUM_OPEN_TXNS);
 
     timeout = MetastoreConf.getTimeVar(conf, ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
     replicationTxnTimeout = MetastoreConf.getTimeVar(conf, ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    retryInterval = MetastoreConf.getTimeVar(conf, ConfVars.HMS_HANDLER_INTERVAL,
-        TimeUnit.MILLISECONDS);
-    retryLimit = MetastoreConf.getIntVar(conf, ConfVars.HMS_HANDLER_ATTEMPTS);
-    deadlockRetryInterval = retryInterval / 10;
     maxOpenTxns = MetastoreConf.getIntVar(conf, ConfVars.MAX_OPEN_TXNS);
-    maxBatchSize = MetastoreConf.getIntVar(conf, ConfVars.JDBC_MAX_BATCH_SIZE);
-
     openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
     
     try {
-      useMinHistoryWriteId = checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
-      
-      // override the config if table does not exists anymore
+      TxnHandlingFeatures.setUseMinHistoryWriteId(checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
+              MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))

Review Comment:
   added snippet of code, we can pass as predicate



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1860004220

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [230 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418805576


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -264,7 +240,7 @@ public void cleanEmptyAbortedAndCommittedTxns() throws MetaException {
      * 2. never deletes the maximum txnId even if it is before the TXN_OPENTXN_TIMEOUT window
      */
     try {
-      long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(jdbcResource.getConnection());
+      long lowWaterMark = jdbcResource.execute(new OpenTxnTimeoutLowBoundaryTxnIdHandler(openTxnTimeOutMillis));

Review Comment:
   should this be a function? how is that different from GenerateCompactionQueueIdFunction?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418694417


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandlingFeatures.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class TxnHandlingFeatures {

Review Comment:
   should it be a singleton instead of class with static fields? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419085985


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),

Review Comment:
   why don't we create `lock` method in txnLockManager?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418864997


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -712,6 +782,7 @@ Set<CompactionInfo> findPotentialCompactions(int abortedThreshold, long abortedT
    * WriteSet tracking is used to ensure proper transaction isolation.  This method deletes the
    * transaction metadata once it becomes unnecessary.
    */
+  @Transactional(POOL_TX)

Review Comment:
   shouldn't this use compaction pool?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418814800


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -546,61 +518,28 @@ public long findMinOpenTxnIdForCleaner() throws MetaException {
   @RetrySemantics.Idempotent
   @Deprecated
   public long findMinTxnIdSeenOpen() {
-    if (!useMinHistoryLevel || useMinHistoryWriteId) {
+    if (!TxnHandlingFeatures.useMinHistoryLevel() || TxnHandlingFeatures.useMinHistoryWriteId()) {
       return Long.MAX_VALUE;
     }
     try {
       Long minId = jdbcResource.getJdbcTemplate().queryForObject("SELECT MIN(\"MHL_MIN_OPEN_TXNID\") FROM \"MIN_HISTORY_LEVEL\"",
           new MapSqlParameterSource(), Long.class);
       return minId == null ? Long.MAX_VALUE : minId;
     } catch (DataAccessException e) {
-      if (e.getCause() instanceof SQLException) {
-        if (dbProduct.isTableNotExistsError((SQLException) e.getCause())) {
-          useMinHistoryLevel = false;
-          return Long.MAX_VALUE;
-        }
+      if (dbProduct.isTableNotExistsError(e)) {
+        TxnHandlingFeatures.setUseMinHistoryLevel(false);
+        return Long.MAX_VALUE;
       }
       LOG.error("Unable to execute findMinTxnIdSeenOpen", e);
       throw e;
     }
   }
 
-  @Override
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType, 

Review Comment:
   won't we need this when we separate Compaction and ACID txn handlers?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418963638


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -514,112 +365,19 @@ public Configuration getConf() {
   @Override
   @RetrySemantics.ReadOnly
   public GetOpenTxnsInfoResponse getOpenTxnsInfo() throws MetaException {
-    return getOpenTxnsList(true).toOpenTxnsInfoResponse();
+    return jdbcResource.execute(new GetOpenTxnsListHandler(true, openTxnTimeOutMillis)).toOpenTxnsInfoResponse();
   }
 
   @Override
   @RetrySemantics.ReadOnly
   public GetOpenTxnsResponse getOpenTxns() throws MetaException {
-    return getOpenTxnsList(false).toOpenTxnsResponse(Arrays.asList(TxnType.READ_ONLY));
-  }
-
-  private GetOpenTxnsResponse getOpenTxns(Connection dbConn) throws MetaException {
-    return getOpenTxnsList(false, dbConn).toOpenTxnsResponse(Arrays.asList(TxnType.READ_ONLY));
+    return jdbcResource.execute(new GetOpenTxnsListHandler(false, openTxnTimeOutMillis)).toOpenTxnsResponse(Collections.singletonList(TxnType.READ_ONLY));

Review Comment:
   too long



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418946961


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -457,7 +306,8 @@ public void setConf(Configuration conf){
       throw new RuntimeException(e);
     }
 
-    sqlRetryHandler = new SqlRetryHandler(conf, jdbcResource.getDatabaseProduct());    
+    sqlRetryHandler = new SqlRetryHandler(conf, jdbcResource.getDatabaseProduct());
+    txnLockHandler = TransactionalRetryProxy.getProxy(new TxnLockHandlerImpl(jdbcResource), sqlRetryHandler, jdbcResource);

Review Comment:
   minor, should we change the order of params?
   ````
   TransactionalRetryProxy.getProxy(sqlRetryHandler, jdbcResource, new TxnLockHandlerImpl(jdbcResource))
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419078718


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {
+    int threshold = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD);
+    MetricsInfo metrics = jdbcResource.execute(MetricsInfoHandler.INSTANCE);
+    Set<String> resourceNames = jdbcResource.execute(new TablesWithAbortedTxnsHandler(threshold));
+    metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
+    metrics.setTablesWithXAbortedTxns(resourceNames);    
+    return metrics;
   }
 
-  /**
-   * Heartbeats on the lock table.  This commits, so do not enter it with any state.
-   * Should not be called on a lock that belongs to transaction.
-   */
-  private void heartbeatLock(Connection dbConn, long extLockId)
-    throws NoSuchLockException, SQLException, MetaException {
-    // If the lock id is 0, then there are no locks in this heartbeat
-    if (extLockId == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String updateHeartbeatQuery = "UPDATE \"HIVE_LOCKS\" SET \"HL_LAST_HEARTBEAT\" = " +
-          getEpochFn(dbProduct) + " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute update <{}>", updateHeartbeatQuery);
-      int rc = stmt.executeUpdate(updateHeartbeatQuery);
-      if (rc < 1) {
-        LOG.error("Failure to update last heartbeat for extLockId={}.", extLockId);
-        dbConn.rollback();
-        throw new NoSuchLockException("No such lock: " + JavaUtils.lockIdToString(extLockId));
-      }
-      LOG.debug("Successfully heartbeated for extLockId={}", extLockId);
-      dbConn.commit();
-    }
-  }
 
-  // Heartbeats on the txn table.  This commits, so do not enter it with any state
-  private void heartbeatTxn(Connection dbConn, long txnid)
-    throws NoSuchTxnException, TxnAbortedException, SQLException, MetaException {
-    // If the txnid is 0, then there are no transactions in this heartbeat
-    if (txnid == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String s = "UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-          " WHERE \"TXN_ID\" = " + txnid + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-      LOG.debug("Going to execute update <{}>", s);
-      int rc = stmt.executeUpdate(s);
-      if (rc < 1) {
-        ensureValidTxn(dbConn, txnid, stmt); // This should now throw some useful exception.
-        LOG.error("Can neither heartbeat txn (txnId={}) nor confirm it as invalid.", txnid);
-        dbConn.rollback();
-        throw new NoSuchTxnException("No such txn: " + txnid);
-      }
-      LOG.debug("Successfully heartbeated for txnId={}", txnid);
-      dbConn.commit();
-    }
+  private static void shouldNeverHappen(long txnid) {

Review Comment:
   remove extra new line above and maybe move all private methods to the end of the class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419073617


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -5785,261 +1068,22 @@ private void timeOutLocks() {
    */
   @RetrySemantics.Idempotent
   public void performTimeOuts() {
-    jdbcResource.bindDataSource(POOL_TX);
-    try (TransactionContext context = jdbcResource.getTransactionManager().getTransaction(PROPAGATION_REQUIRED)) {
-      //We currently commit after selecting the TXNS to abort.  So whether SERIALIZABLE
-      //READ_COMMITTED, the effect is the same.  We could use FOR UPDATE on Select from TXNS
-      //and do the whole performTimeOuts() in a single huge transaction, but the only benefit
-      //would be to make sure someone cannot heartbeat one of these txns at the same time.
-      //The attempt to heartbeat would block and fail immediately after it's unblocked.
-      //With current (RC + multiple txns) implementation it is possible for someone to send
-      //heartbeat at the very end of the expiry interval, and just after the Select from TXNS
-      //is made, in which case heartbeat will succeed but txn will still be Aborted.
-      //Solving this corner case is not worth the perf penalty.  The client should heartbeat in a
-      //timely way.
-      timeOutLocks();
-      while (true) {
-        String s = " \"TXN_ID\" FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN +
-            " AND (" +
-            "\"TXN_TYPE\" != " + TxnType.REPL_CREATED.getValue() +
-            " AND \"TXN_LAST_HEARTBEAT\" <  " + getEpochFn(dbProduct) + "-" + timeout +
-            " OR " +
-            " \"TXN_TYPE\" = " + TxnType.REPL_CREATED.getValue() +
-            " AND \"TXN_LAST_HEARTBEAT\" <  " + getEpochFn(dbProduct) + "-" + replicationTxnTimeout +
-            ")";
-        //safety valve for extreme cases
-        s = sqlGenerator.addLimitClause(10 * TIMED_OUT_TXN_ABORT_BATCH_SIZE, s);
-
-        LOG.debug("Going to execute query <{}>", s);
-        List<List<Long>> timedOutTxns = jdbcResource.getJdbcTemplate().query(s, rs -> {
-          List<List<Long>> txnbatch = new ArrayList<>();
-          List<Long> currentBatch = new ArrayList<>(TIMED_OUT_TXN_ABORT_BATCH_SIZE);
-          while (rs.next()) {
-            currentBatch.add(rs.getLong(1));
-            if (currentBatch.size() == TIMED_OUT_TXN_ABORT_BATCH_SIZE) {
-              txnbatch.add(currentBatch);
-              currentBatch = new ArrayList<>(TIMED_OUT_TXN_ABORT_BATCH_SIZE);
-            }
-          }
-          if (currentBatch.size() > 0) {
-            txnbatch.add(currentBatch);
-          }
-          return txnbatch;
-        });
-        //noinspection DataFlowIssue
-        if (timedOutTxns.size() == 0) {
-          jdbcResource.getTransactionManager().commit(context);
-          return;
-        }
-
-        Object savePoint = context.getTransactionStatus().createSavepoint();
-
-        int numTxnsAborted = 0;
-        for (List<Long> batchToAbort : timedOutTxns) {
-          context.getTransactionStatus().releaseSavepoint(savePoint);
-          savePoint = context.getTransactionStatus().createSavepoint();
-          if (abortTxns(jdbcResource.getConnection(), batchToAbort, true, false, false, TxnErrorMsg.ABORT_TIMEOUT) == batchToAbort.size()) {
-            numTxnsAborted += batchToAbort.size();
-            //todo: add TXNS.COMMENT filed and set it to 'aborted by system due to timeout'
-          } else {
-            //could not abort all txns in this batch - this may happen because in parallel with this
-            //operation there was activity on one of the txns in this batch (commit/abort/heartbeat)
-            //This is not likely but may happen if client experiences long pause between heartbeats or
-            //unusually long/extreme pauses between heartbeat() calls and other logic in checkLock(),
-            //lock(), etc.
-            context.getTransactionStatus().rollbackToSavepoint(savePoint);
-          }
-        }
-        LOG.info("Aborted {} transaction(s) due to timeout", numTxnsAborted);
-        if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-          Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_TIMED_OUT_TXNS).inc(numTxnsAborted);
-        }
-      }
-    } catch (MetaException | SQLException e) {
-      LOG.warn("Aborting timed out transactions failed due to " + e.getMessage(), e);
-    } finally {
-      jdbcResource.unbindDataSource();
-    }
+    new PerformTimeoutsFunction(timeout, replicationTxnTimeout).execute(jdbcResource);
   }
 
   @Override
   @RetrySemantics.ReadOnly
   public void countOpenTxns() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        String s = "SELECT COUNT(*) FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN;
-        LOG.debug("Going to execute query <{}>", s);
-        rs = stmt.executeQuery(s);
-        if (!rs.next()) {
-          LOG.error("Transaction database not properly configured, can't find txn_state from TXNS.");
-        } else {
-          Long numOpen = rs.getLong(1);
-          if (numOpen > Integer.MAX_VALUE) {
-            LOG.error("Open transaction count above {}, can't count that high!", Integer.MAX_VALUE);
-          } else {
-            numOpenTxns.set(numOpen.intValue());
-          }
-        }
-      } catch (SQLException e) {
-        LOG.info("Failed to update number of open transactions");
-        checkRetryable(e, "countOpenTxns()");
-      } finally {
-        close(rs, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      countOpenTxns();
-    }
-  }
-
-  /**
-   * Add min history level entry for each generated txn record
-   * @param dbConn Connection
-   * @param txnIds new transaction ids
-   * @deprecated Remove this method when min_history_level table is dropped
-   * @throws SQLException ex
-   */
-  @Deprecated
-  private void addTxnToMinHistoryLevel(Connection dbConn, List<Long> txnIds, long minOpenTxnId) throws SQLException {
-    if (!useMinHistoryLevel) {
-      return;
-    }
-    // Need to register minimum open txnid for current transactions into MIN_HISTORY table.
-    try (Statement stmt = dbConn.createStatement()) {
-      List<String> rows = txnIds.stream().map(txnId -> txnId + ", " + minOpenTxnId).collect(Collectors.toList());
-
-      // Insert transaction entries into MIN_HISTORY_LEVEL.
-      List<String> inserts =
-          sqlGenerator.createInsertValuesStmt("\"MIN_HISTORY_LEVEL\" (\"MHL_TXNID\", \"MHL_MIN_OPEN_TXNID\")", rows);
-      for (String insert : inserts) {
-        LOG.debug("Going to execute insert <{}>", insert);
-        stmt.execute(insert);
-      }
-      LOG.info("Added entries to MIN_HISTORY_LEVEL for current txns: ({}) with min_open_txn: {}", txnIds, minOpenTxnId);
-    } catch (SQLException e) {
-      if (dbProduct.isTableNotExistsError(e)) {
-        // If the table does not exists anymore, we disable the flag and start to work the new way
-        // This enables to switch to the new functionality without a restart
-        useMinHistoryLevel = false;
-      } else {
-        throw e;
-      }
+    int openTxns = jdbcResource.execute(new CountOpenTxnsHandler());
+    if (openTxns > -1) {
+      numOpenTxns.set(openTxns);
     }
   }
 
   @Override
   @RetrySemantics.SafeToRetry

Review Comment:
   should we keep this? please check for othe methods as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419194932


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ConditionalCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import org.apache.hadoop.hive.metastore.DatabaseProduct;
+
+/**
+ * {@link ParameterizedCommand}. {@link ParameterizedBatchCommand}, and {@link InClauseBatchCommand} implementations can also
+ * implement this interface, marking that the execution is conditial, and the command wants to get notified about 
+ * execution errors. Can be used to implement commands depending on some feature flag(s).  
+ */
+public interface ConditionalCommand {
+
+  /**
+   * Indicates if the command should be executed or not
+   * @param databaseProduct
+   * @return Returns true if the command can be executed, false otherwise.
+   */
+  boolean shouldBeUsed(DatabaseProduct databaseProduct);

Review Comment:
   ParameterizedCommand, what do you mean by "execution is conditional"? do we need some command that shouldn't be executed? what is the usecase?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419107740


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +182,141 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+  public static class TxnHandlingFeatures {    

Review Comment:
   please rename to ConfVars. 
   ````
   private static final ConfVars confVars;
   
   confVars = new ConfVars(table->checkIfTableIsUsable(table), conf)
   
   public static final ConfVars {
     private boolean useMinHistoryLevel;
     private boolean useMinHistoryWriteId;
     
     public ConfVars(Predicate<String> tableCheck, Conf conf){
       useMinHistoryLevel = tableCheck.test("MIN_HISTORY_WRITE_ID", 
     MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))
     }
   }
   ````
   
   TxnHandler.getConfVars().useMinHistoryWriteId
   TxnHandler.getConfVar().useMinHistoryLevel



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422557800


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +182,141 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+  public static class TxnHandlingFeatures {    

Review Comment:
   done, except I did not remove the setters and getters as they are used outside of `TxnHandler` too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425281789


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),

Review Comment:
   `enqueuelock` also has `SqlRetry` and `Transactional` annotations so it needs to be on the interface and called through the proxy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425248487


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,260 +183,171 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
-  
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
 
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
+  
+  public static class ConfVars {    

Review Comment:
   final + private constructor



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425295561


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/commands/AddWriteIdsToMinHistoryCommand.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.commands;

Review Comment:
   ok, done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1855881814

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [215 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1399122432


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ConditionalCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import org.apache.hadoop.hive.metastore.DatabaseProduct;
+
+/**
+ * {@link ParameterizedCommand}. {@link ParameterizedBatchCommand}, and {@link InClauseBatchCommand} implementations can also
+ * implement this interface, marking that the execution is conditial, and the command wants to get notified about 
+ * execution errors. Can be used to implement commands depending on some feature flag(s).  
+ */
+public interface ConditionalCommand {
+
+  /**
+   * Indicates if the command should be executed or not
+   * @param databaseProduct
+   * @return Returns true if the command can be executed, false otherwise.
+   */
+  boolean shouldUse(DatabaseProduct databaseProduct);

Review Comment:
   Ok, renamed it to `shouldBeUsed()` it sounds better for me too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1759746423

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [15 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [160 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1816150900

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [18 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [168 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1834353080

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [18 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [169 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1829698849

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [18 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [168 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1841106819

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [161 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418805576


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -264,7 +240,7 @@ public void cleanEmptyAbortedAndCommittedTxns() throws MetaException {
      * 2. never deletes the maximum txnId even if it is before the TXN_OPENTXN_TIMEOUT window
      */
     try {
-      long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(jdbcResource.getConnection());
+      long lowWaterMark = jdbcResource.execute(new OpenTxnTimeoutLowBoundaryTxnIdHandler(openTxnTimeOutMillis));

Review Comment:
   should this be a function? how is that different from GenerateCompactionQueueIdFunction?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418821749


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -546,61 +518,28 @@ public long findMinOpenTxnIdForCleaner() throws MetaException {
   @RetrySemantics.Idempotent
   @Deprecated
   public long findMinTxnIdSeenOpen() {
-    if (!useMinHistoryLevel || useMinHistoryWriteId) {
+    if (!TxnHandlingFeatures.useMinHistoryLevel() || TxnHandlingFeatures.useMinHistoryWriteId()) {
       return Long.MAX_VALUE;
     }
     try {
       Long minId = jdbcResource.getJdbcTemplate().queryForObject("SELECT MIN(\"MHL_MIN_OPEN_TXNID\") FROM \"MIN_HISTORY_LEVEL\"",
           new MapSqlParameterSource(), Long.class);
       return minId == null ? Long.MAX_VALUE : minId;
     } catch (DataAccessException e) {
-      if (e.getCause() instanceof SQLException) {
-        if (dbProduct.isTableNotExistsError((SQLException) e.getCause())) {
-          useMinHistoryLevel = false;
-          return Long.MAX_VALUE;
-        }
+      if (dbProduct.isTableNotExistsError(e)) {
+        TxnHandlingFeatures.setUseMinHistoryLevel(false);
+        return Long.MAX_VALUE;
       }
       LOG.error("Unable to execute findMinTxnIdSeenOpen", e);
       throw e;
     }
   }
 
-  @Override
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType, 

Review Comment:
   A follow up task has already been created for it. IT would made this PR even larger so I did not do it now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422528982


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +431,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table
+    StopWatch generateTransactionWatch = new StopWatch();
+    generateTransactionWatch.start();
 
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        generateTransactionWatch.stop();
-        long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
-        TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-        if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
-          /*
-           * The commit was too slow, we can not allow this to continue (except if it is read only,
-           * since that can not cause dirty reads).
-           * When calculating the snapshot for a given transaction, we look back for possible open transactions
-           * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
-           * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
-           * because there can be other transactions running, that didn't considered this transactionId open,
-           * this could cause dirty reads.
-           */
-          LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
-          deleteInvalidOpenTransactions(dbConn, txnIds);
-          dbConn.commit();
-          /*
-           * We do not throw RetryException directly, to not circumvent the max retry limit
-           */
-          throw new SQLException("OpenTxnTimeOut exceeded", MANUAL_RETRY);
-        }
-        return new OpenTxnsResponse(txnIds);
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "openTxns(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return openTxns(rqst);
-    }
-  }
+    List<Long> txnIds = new OpenTxnsFunction(rqst, openTxnTimeOutMillis, transactionalListeners).execute(jdbcResource);
 
-  private List<Long> openTxns(Connection dbConn, OpenTxnRequest rqst)
-          throws SQLException, MetaException {
-    int numTxns = rqst.getNum_txns();
-    // Make sure the user has not requested an insane amount of txns.
-    int maxTxns = MetastoreConf.getIntVar(conf, ConfVars.TXN_MAX_OPEN_BATCH);
-    if (numTxns > maxTxns) {
-      numTxns = maxTxns;
-    }
-    List<PreparedStatement> insertPreparedStmts = null;
+    LOG.debug("Going to commit");
+    jdbcResource.getTransactionManager().getActiveTransaction().createSavepoint();

Review Comment:
   It was there:
   
   https://github.com/apache/hive/pull/4566/commits/f7974d8f6f365fe244ccc7e75da5f9ab5d85fc7a#diff-ccc50cf147261af64a10e20359f9c52f1d4797bc8e03e682d6a88891ee941f02L699



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1850372416

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [205 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418694417


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandlingFeatures.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class TxnHandlingFeatures {

Review Comment:
   should it be a singleton instead of class with static fields? + would be better to have it nested in TxnHandler 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418851314


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandlingFeatures.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class TxnHandlingFeatures {

Review Comment:
   we can make it public



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418938461


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +181,115 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockHandler txnLockHandler;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf,
+                  MetastoreConf.getIntVar(conf, ConfVars.HIVE_COMPACTOR_CONNECTION_POOLING_MAX_CONNECTIONS));
+            }            
+          }
+          if (dbProduct == null) {
+            try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPool)) {
+              determineDatabaseProduct(dbConn);
+            } catch (SQLException e) {
+              LOG.error("Unable to determine database product", e);
+              throw new RuntimeException(e);
+            }
+          }
+          if (sqlGenerator == null) {
+            sqlGenerator = new SQLGenerator(dbProduct, conf);
+          }
+          initialized = true;
         }
       }
+    }
 
-      if (sqlGenerator == null) {
-        sqlGenerator = new SQLGenerator(dbProduct, conf);
-      }
-      
-      if (jdbcResource == null) {
-        jdbcResource = new MultiDataSourceJdbcResource(dbProduct);
-        jdbcResource.registerDataSource(POOL_TX, connPool);
-        jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
-      }      
+    if (jdbcResource == null) {
+      jdbcResource = new MultiDataSourceJdbcResource(dbProduct, conf, sqlGenerator);
+      jdbcResource.registerDataSource(POOL_TX, connPool);
+      jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
+      jdbcResource.registerDataSource(POOL_COMPACTOR, connPoolCompactor);
     }
+    
+    mutexAPI = new HiveMutex(sqlGenerator, jdbcResource);
 
     numOpenTxns = Metrics.getOrCreateGauge(MetricsConstants.NUM_OPEN_TXNS);
 
     timeout = MetastoreConf.getTimeVar(conf, ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
     replicationTxnTimeout = MetastoreConf.getTimeVar(conf, ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    retryInterval = MetastoreConf.getTimeVar(conf, ConfVars.HMS_HANDLER_INTERVAL,
-        TimeUnit.MILLISECONDS);
-    retryLimit = MetastoreConf.getIntVar(conf, ConfVars.HMS_HANDLER_ATTEMPTS);
-    deadlockRetryInterval = retryInterval / 10;
     maxOpenTxns = MetastoreConf.getIntVar(conf, ConfVars.MAX_OPEN_TXNS);
-    maxBatchSize = MetastoreConf.getIntVar(conf, ConfVars.JDBC_MAX_BATCH_SIZE);
-
     openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
     
     try {
-      useMinHistoryWriteId = checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
-      
-      // override the config if table does not exists anymore
+      TxnHandlingFeatures.setUseMinHistoryWriteId(checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
+              MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))

Review Comment:
   formatting, extra spaces



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418936538


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +180,114 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;

Review Comment:
   but as we discussed, we plan to separate TxnHandler and CompTxnHandler, moving comp pool initialization here is not what we want to accomplish 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419107740


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +182,141 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+  public static class TxnHandlingFeatures {    

Review Comment:
   please rename to ConfVars. 
   ````
   private static final ConfVars vars;
   
   vars = new ConfVars(table->checkIfTableIsUsable(table), conf)
   
   public static final ConfVars {
     private boolean useMinHistoryLevel;
     private boolean useMinHistoryWriteId;
     
     public ConfVars(Predicate<String> tableCheck, Conf conf){
       useMinHistoryLevel = tableCheck.test("MIN_HISTORY_WRITE_ID", 
     MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))
     }
   }
   ````
   
   TxnHandler.getConfVars().useMinHistoryWriteId
   TxnHandler.getConfVar().useMinHistoryLevel



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418694417


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandlingFeatures.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class TxnHandlingFeatures {

Review Comment:
   should it be a singleton instead of class with static fields? + would be better to have it nested in TxnHandler .Features



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418837129


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          } else if (!context.isCompleted()) {
+            jdbcResource.getTransactionManager().commit(context);
+          }
           return result;
+        } catch (ProgrammaticRollbackException e) {
+          if (context != null && !context.isCompleted()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          }          
+          return e.getResult();
         } catch (Exception e) {
-          if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
-              Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
-            throw e;
-          }
           if (context != null) {
-            if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {
+            if (transactional.noRollbackFor().length > 0 || transactional.noRollbackForClassName().length > 0) {
+              if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
+                  Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
+                jdbcResource.getTransactionManager().commit(context);
+              } else {
+                jdbcResource.getTransactionManager().rollback(context);
+              }
+            } else if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {

Review Comment:
   The previous is **no**RollbackFor() and **no**RollbackForClassName()



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          } else if (!context.isCompleted()) {
+            jdbcResource.getTransactionManager().commit(context);
+          }
           return result;
+        } catch (ProgrammaticRollbackException e) {
+          if (context != null && !context.isCompleted()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          }          
+          return e.getResult();
         } catch (Exception e) {
-          if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
-              Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
-            throw e;
-          }
           if (context != null) {
-            if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {
+            if (transactional.noRollbackFor().length > 0 || transactional.noRollbackForClassName().length > 0) {
+              if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
+                  Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
+                jdbcResource.getTransactionManager().commit(context);
+              } else {
+                jdbcResource.getTransactionManager().rollback(context);
+              }
+            } else if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {

Review Comment:
   The previous is **!no!**RollbackFor() and **!no!**RollbackForClassName()



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1764852806

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [15 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [152 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425248487


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,260 +183,171 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
-  
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
 
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
+  
+  public static class ConfVars {    

Review Comment:
   final



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425296504


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import java.util.Stack;
+
+public class StackThreadLocal<T> {

Review Comment:
   if we can use `counter` instead in SqlRetryHandler, this could be embedded in MultiDataSourceJdbcResource



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425299454


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/functions/HeartBeatLockFunction.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.functions;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionalFunction;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnUtils.getEpochFn;
+
+/**
+ * Heartbeats on the lock table.  This commits, so do not enter it with any state.
+ * Should not be called on a lock that belongs to transaction.
+ */
+public class HeartBeatLockFunction implements TransactionalFunction<Void> {

Review Comment:
   fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1749286785

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [10 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [131 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1747024345

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [9 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [117 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418819741


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import java.util.Stack;
+
+public class StackThreadLocal<T> {
+
+  private final ThreadLocal<Stack<T>> threadLocal = new ThreadLocal<>();
+
+  public void set(T value) {
+    Stack<T> stack = threadLocal.get();
+    if (stack == null) {
+      stack = new Stack<>();

Review Comment:
   use Deque over Stack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418842387


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          } else if (!context.isCompleted()) {
+            jdbcResource.getTransactionManager().commit(context);
+          }
           return result;
+        } catch (ProgrammaticRollbackException e) {
+          if (context != null && !context.isCompleted()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          }          
+          return e.getResult();
         } catch (Exception e) {
-          if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
-              Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
-            throw e;
-          }
           if (context != null) {
-            if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {
+            if (transactional.noRollbackFor().length > 0 || transactional.noRollbackForClassName().length > 0) {
+              if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
+                  Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
+                jdbcResource.getTransactionManager().commit(context);
+              } else {
+                jdbcResource.getTransactionManager().rollback(context);
+              }
+            } else if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {

Review Comment:
   yea, sorry, removed
   is there an order to check for `noRollbackFor` and `rollbackFor`? could we get different outcome?
   is that possible to extract this if-else logic into a method `shouldRollback()` or something?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418845605


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -149,15 +164,23 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
           LOG.debug("Successfull method invocation within retry context: {}", callerId);
           return result;
         } catch (IllegalAccessException | InvocationTargetException | UndeclaredThrowableException e) {
-          if (e.getCause() instanceof MetaException) {
-            throw (MetaException) e.getCause();
+          if (e.getCause() instanceof TException) {
+            throw (TException) e.getCause();
           } else if (e.getCause() instanceof RuntimeException) {
             throw (RuntimeException) e.getCause();
           } else {
             throw new RuntimeException(e);
           }
-        } catch (Throwable e) {
-          throw new RuntimeException(e);
+        } catch (TException | DataAccessException e) {

Review Comment:
   should we extract to rethrowException(e), do we use same thing somewhere else? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418875205


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/entities/LockInfo.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.entities;
+
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.hadoop.hive.metastore.api.LockState;
+import org.apache.hadoop.hive.metastore.api.LockType;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.ShowLocksResponseElement;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.apache.hadoop.hive.metastore.utils.LockTypeUtil;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+
+public class LockInfo {
+
+  public static final char LOCK_ACQUIRED = 'a';
+  public static final char LOCK_WAITING = 'w';
+  
+
+  private final long extLockId;
+  private final long intLockId;
+  //0 means there is no transaction, i.e. it a select statement which is not part of
+  //explicit transaction or a IUD statement that is not writing to ACID table
+  private final long txnId;
+  private final String db;
+  private final String table;
+  private final String partition;
+  private final LockState state;
+  private final LockType type;
+
+  // Assumes the result set is set to a valid row
+  public LockInfo(ResultSet rs) throws SQLException, MetaException {

Review Comment:
   not sure it's a good pattern to pass ResultSet into constructor.  I think it's better to use Builder, but I'll leave it to you



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418912702


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/functions/HeartBeatTxnFunction.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.functions;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.txn.TxnStatus;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionalFunction;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.sql.Types;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnUtils.getEpochFn;
+
+/**
+ * Heartbeats on the txn table. This commits, so do not enter it with any state. 
+ */
+public class HeartBeatTxnFunction implements TransactionalFunction<Void> {

Review Comment:
   HeartbeatTxnFunction



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418938461


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +181,115 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
 abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
-
-  // Lock states
-  static final protected char LOCK_ACQUIRED = 'a';
-  static final protected char LOCK_WAITING = 'w';
-
-  private static final int ALLOWED_REPEATED_DEADLOCKS = 10;
   private static final Logger LOG = LoggerFactory.getLogger(TxnHandler.class.getName());
 
-  private static DataSource connPool;
-  private static DataSource connPoolMutex;
-
-  private static final String MANUAL_RETRY = "ManualRetry";
-
-  // Query definitions
-  private static final String HIVE_LOCKS_INSERT_QRY = "INSERT INTO \"HIVE_LOCKS\" ( " +
-      "\"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", " +
-      "\"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_USER\", \"HL_HOST\", \"HL_AGENT_INFO\") " +
-      "VALUES (?, ?, ?, ?, ?, ?, ?, ?, %s, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"TXN_COMPONENTS\" (" +
-      "\"TC_TXNID\", \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_OPERATION_TYPE\", \"TC_WRITEID\")" +
-      " VALUES (?, ?, ?, ?, ?, ?)";
-  private static final String TXN_COMPONENTS_DP_DELETE_QUERY = "DELETE FROM \"TXN_COMPONENTS\" " +
-      "WHERE \"TC_TXNID\" = ? AND \"TC_DATABASE\" = ? AND \"TC_TABLE\" = ? AND \"TC_PARTITION\" IS NULL";
-  private static final String INCREMENT_NEXT_LOCK_ID_QUERY = "UPDATE \"NEXT_LOCK_ID\" SET \"NL_NEXT\" = %s";
-  private static final String UPDATE_HIVE_LOCKS_EXT_ID_QUERY = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_EXT_ID\" = %s " +
-      "WHERE \"HL_LOCK_EXT_ID\" = %s";
-  private static final String SELECT_WRITE_ID_QUERY = "SELECT \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE" +
-      " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_TXNID\" = ?";
-  private static final String COMPL_TXN_COMPONENTS_INSERT_QUERY = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" " +
-      "(\"CTC_TXNID\"," + " \"CTC_DATABASE\", \"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\")" +
-      " VALUES (%s, ?, ?, ?, ?, %s)";
-  private static final String TXNS_INSERT_QRY = "INSERT INTO \"TXNS\" " +
-      "(\"TXN_STATE\", \"TXN_STARTED\", \"TXN_LAST_HEARTBEAT\", \"TXN_USER\", \"TXN_HOST\", \"TXN_TYPE\") " +
-      "VALUES(?,%s,%s,?,?,?)";
-  private static final String SELECT_LOCKS_FOR_LOCK_ID_QUERY = "SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", " +
-      "\"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\", \"HL_TXNID\" " +
-      "FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = ?";
-  private static final String SELECT_TIMED_OUT_LOCKS_QUERY = "SELECT DISTINCT \"HL_LOCK_EXT_ID\" FROM \"HIVE_LOCKS\" " +
-      "WHERE \"HL_LAST_HEARTBEAT\" < %s - :timeout AND \"HL_TXNID\" = 0";
-  private static final String TXN_TO_WRITE_ID_INSERT_QUERY = "INSERT INTO \"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", " +
-      "\"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String MIN_HISTORY_WRITE_ID_INSERT_QUERY = "INSERT INTO \"MIN_HISTORY_WRITE_ID\" (\"MH_TXNID\", " +
-      "\"MH_DATABASE\", \"MH_TABLE\", \"MH_WRITEID\") VALUES (?, ?, ?, ?)";
-  private static final String SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID =
-      "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-  private static final String SELECT_METRICS_INFO_QUERY =
-      "SELECT * FROM (SELECT COUNT(*) FROM \"TXN_TO_WRITE_ID\") \"TTWID\" CROSS JOIN (" +
-      "SELECT COUNT(*) FROM \"COMPLETED_TXN_COMPONENTS\") \"CTC\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" = "+ TxnType.REPL_CREATED.getValue() +") \"TR\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.OPEN + "' AND \"TXN_TYPE\" != "+ TxnType.REPL_CREATED.getValue() +") \"T\" CROSS JOIN (" +
-      "SELECT COUNT(*), MIN(\"TXN_ID\"), ({0} - MIN(\"TXN_STARTED\"))/1000 FROM \"TXNS\" WHERE \"TXN_STATE\"='" +
-        TxnStatus.ABORTED + "') \"A\" CROSS JOIN (" +
-      "SELECT COUNT(*), ({0} - MIN(\"HL_ACQUIRED_AT\"))/1000 FROM \"HIVE_LOCKS\") \"HL\" CROSS JOIN (" +
-      "SELECT ({0} - MIN(\"CQ_COMMIT_TIME\"))/1000 from \"COMPACTION_QUEUE\" WHERE " +
-          "\"CQ_STATE\"=''" + READY_FOR_CLEANING + "'') OLDEST_CLEAN";
-  private static final String SELECT_TABLES_WITH_X_ABORTED_TXNS =
-      "SELECT \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" FROM \"TXN_COMPONENTS\" " +
-          "INNER JOIN \"TXNS\" ON \"TC_TXNID\" = \"TXN_ID\" WHERE \"TXN_STATE\" = " + TxnStatus.ABORTED +
-      " GROUP BY \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\" HAVING COUNT(\"TXN_ID\") > ?";
-  
-  private static final String  EXCL_CTAS_ERR_MSG = 
-      "Failed to initiate a concurrent CTAS operation with the same table name, lockInfo : %s";
-  private static final String ZERO_WAIT_READ_ERR_MSG = "Unable to acquire read lock due to an existing exclusive lock {%s}";
-
-
-  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
-
   // Maximum number of open transactions that's allowed
   private static volatile int maxOpenTxns = 0;
   // Whether number of open transactions reaches the threshold
   private static volatile boolean tooManyOpenTxns = false;
+  // Current number of open txns
+  private static AtomicInteger numOpenTxns;
+
+  private static volatile boolean initialized = false;
+  private static DataSource connPool;
+  private static DataSource connPoolMutex;
+  protected static DataSource connPoolCompactor;
+
+  protected static DatabaseProduct dbProduct;
+  protected static SQLGenerator sqlGenerator;
+  protected static long openTxnTimeOutMillis;
 
   /**
    * Number of consecutive deadlocks we have seen
    */
-  private int deadlockCnt;
-  private long deadlockRetryInterval;
   protected Configuration conf;
-  protected static DatabaseProduct dbProduct;
-  protected static SQLGenerator sqlGenerator;
-  private static long openTxnTimeOutMillis;
 
+  protected List<TransactionalMetaStoreEventListener> transactionalListeners;
   // (End user) Transaction timeout, in milliseconds.
   private long timeout;
   private long replicationTxnTimeout;
 
-  private int maxBatchSize;
-  private String identifierQuoteString; // quotes to use for quoting tables, where necessary
-  private long retryInterval;
-  private int retryLimit;
-  private int retryNum;
-  // Current number of open txns
-  private AtomicInteger numOpenTxns;
-  // Whether to use min_history_level table or not.
-  // At startup we read it from the config, but set it to false if min_history_level does nto exists.
-  static boolean useMinHistoryLevel;
-  static boolean useMinHistoryWriteId;
-
-  private static SqlRetryHandler sqlRetryHandler;
-  protected static MultiDataSourceJdbcResource jdbcResource;
+  private MutexAPI mutexAPI;
+  private TxnLockHandler txnLockHandler;
+  private SqlRetryHandler sqlRetryHandler;
+  protected MultiDataSourceJdbcResource jdbcResource;
 
-  /**
-   * Derby specific concurrency control
-   */
-  private static final ReentrantLock derbyLock = new ReentrantLock(true);
-  /**
-   * must be static since even in UT there may be > 1 instance of TxnHandler
-   * (e.g. via Compactor services)
-   */
-  private final static ConcurrentHashMap<String, Semaphore> derbyKey2Lock = new ConcurrentHashMap<>();
   private static final String hostname = JavaUtils.hostname();
 
-  // Private methods should never catch SQLException and then throw MetaException.  The public
-  // methods depend on SQLException coming back so they can detect and handle deadlocks.  Private
-  // methods should only throw MetaException when they explicitly know there's a logic error and
-  // they want to throw past the public methods.
-  //
-  // All public methods that write to the database have to check for deadlocks when a SQLException
-  // comes back and handle it if they see one.  This has to be done with the connection pooling
-  // in mind.  To do this they should call checkRetryable() AFTER rolling back the db transaction,
-  // and then they should catch RetryException and call themselves recursively. See commitTxn for an example.
-
   public TxnHandler() {
   }
 
   /**
    * This is logically part of c'tor and must be called prior to any other method.
    * Not physically part of c'tor due to use of reflection
    */
-  public void setConf(Configuration conf){
+  public void setConf(Configuration conf) {
     this.conf = conf;
 
-    int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
-    synchronized (TxnHandler.class) {
-      try (DataSourceProvider.DataSourceNameConfigurator configurator =
-               new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
-        if (connPool == null) {
-          connPool = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-        if (connPoolMutex == null) {
-          configurator.resetName(POOL_MUTEX);
-          connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
-        }
-      }
-      if (dbProduct == null) {
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          determineDatabaseProduct(dbConn);
-        } catch (SQLException e) {
-          LOG.error("Unable to determine database product", e);
-          throw new RuntimeException(e);
+    if (!initialized) {
+      synchronized (TxnHandler.class) {
+        if (!initialized) {
+          try (DataSourceProvider.DataSourceNameConfigurator configurator =
+                   new DataSourceProvider.DataSourceNameConfigurator(conf, POOL_TX)) {
+            int maxPoolSize = MetastoreConf.getIntVar(conf, ConfVars.CONNECTION_POOLING_MAX_CONNECTIONS);
+            if (connPool == null) {
+              connPool = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolMutex == null) {
+              configurator.resetName(POOL_MUTEX);
+              connPoolMutex = setupJdbcConnectionPool(conf, maxPoolSize);
+            }
+            if (connPoolCompactor == null) {
+              configurator.resetName(POOL_COMPACTOR);
+              connPoolCompactor = setupJdbcConnectionPool(conf,
+                  MetastoreConf.getIntVar(conf, ConfVars.HIVE_COMPACTOR_CONNECTION_POOLING_MAX_CONNECTIONS));
+            }            
+          }
+          if (dbProduct == null) {
+            try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED, connPool)) {
+              determineDatabaseProduct(dbConn);
+            } catch (SQLException e) {
+              LOG.error("Unable to determine database product", e);
+              throw new RuntimeException(e);
+            }
+          }
+          if (sqlGenerator == null) {
+            sqlGenerator = new SQLGenerator(dbProduct, conf);
+          }
+          initialized = true;
         }
       }
+    }
 
-      if (sqlGenerator == null) {
-        sqlGenerator = new SQLGenerator(dbProduct, conf);
-      }
-      
-      if (jdbcResource == null) {
-        jdbcResource = new MultiDataSourceJdbcResource(dbProduct);
-        jdbcResource.registerDataSource(POOL_TX, connPool);
-        jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
-      }      
+    if (jdbcResource == null) {
+      jdbcResource = new MultiDataSourceJdbcResource(dbProduct, conf, sqlGenerator);
+      jdbcResource.registerDataSource(POOL_TX, connPool);
+      jdbcResource.registerDataSource(POOL_MUTEX, connPoolMutex);
+      jdbcResource.registerDataSource(POOL_COMPACTOR, connPoolCompactor);
     }
+    
+    mutexAPI = new HiveMutex(sqlGenerator, jdbcResource);
 
     numOpenTxns = Metrics.getOrCreateGauge(MetricsConstants.NUM_OPEN_TXNS);
 
     timeout = MetastoreConf.getTimeVar(conf, ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
     replicationTxnTimeout = MetastoreConf.getTimeVar(conf, ConfVars.REPL_TXN_TIMEOUT, TimeUnit.MILLISECONDS);
-    retryInterval = MetastoreConf.getTimeVar(conf, ConfVars.HMS_HANDLER_INTERVAL,
-        TimeUnit.MILLISECONDS);
-    retryLimit = MetastoreConf.getIntVar(conf, ConfVars.HMS_HANDLER_ATTEMPTS);
-    deadlockRetryInterval = retryInterval / 10;
     maxOpenTxns = MetastoreConf.getIntVar(conf, ConfVars.MAX_OPEN_TXNS);
-    maxBatchSize = MetastoreConf.getIntVar(conf, ConfVars.JDBC_MAX_BATCH_SIZE);
-
     openTxnTimeOutMillis = MetastoreConf.getTimeVar(conf, ConfVars.TXN_OPENTXN_TIMEOUT, TimeUnit.MILLISECONDS);
     
     try {
-      useMinHistoryWriteId = checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
-        MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID));
-      
-      // override the config if table does not exists anymore
+      TxnHandlingFeatures.setUseMinHistoryWriteId(checkIfTableIsUsable("MIN_HISTORY_WRITE_ID", 
+              MetastoreConf.getBoolVar(conf, ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))

Review Comment:
   formatting, extra spaces
   btw, can't we move that logic checkIfTableIsUsable inside of `TxnHandler.ConfVars` init or something



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419068628


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -6052,450 +1096,53 @@ protected synchronized static DataSource setupJdbcConnectionPool(Configuration c
         throw new RuntimeException(e);
       }
     } else {
-      String connectionPooler = MetastoreConf.getVar(conf, ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
+      String connectionPooler = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
       if ("none".equals(connectionPooler)) {
         LOG.info("Choosing not to pool JDBC connections");
-        return new NoPoolConnectionPool(conf);
+        return new NoPoolConnectionPool(conf, dbProduct);
       } else {
         throw new RuntimeException("Unknown JDBC connection pooling " + connectionPooler);
       }
     }
   }
 
-  /**
-   * Returns true if {@code ex} should be retried
-   */
-  static boolean isRetryable(Configuration conf, Exception ex) {
-    if(ex instanceof SQLException) {
-      SQLException sqlException = (SQLException)ex;
-      if (MANUAL_RETRY.equalsIgnoreCase(sqlException.getSQLState())) {
-        // Manual retry exception was thrown
-        return true;
-      }
-      if ("08S01".equalsIgnoreCase(sqlException.getSQLState())) {
-        //in MSSQL this means Communication Link Failure
-        return true;
-      }
-      if ("ORA-08176".equalsIgnoreCase(sqlException.getSQLState()) ||
-        sqlException.getMessage().contains("consistent read failure; rollback data not available")) {
-        return true;
-      }
-
-      String regex = MetastoreConf.getVar(conf, ConfVars.TXN_RETRYABLE_SQLEX_REGEX);
-      if (regex != null && !regex.isEmpty()) {
-        String[] patterns = regex.split(",(?=\\S)");
-        String message = getMessage((SQLException)ex);
-        for (String p : patterns) {
-          if (Pattern.matches(p, message)) {
-            return true;
-          }
-        }
-      }
-      //see also https://issues.apache.org/jira/browse/HIVE-9938
-    }
-    return false;
-  }
-
-  private boolean isDuplicateKeyError(SQLException ex) {
-    return dbProduct.isDuplicateKeyError(ex);
-  }
-
-  private static String getMessage(SQLException ex) {
-    return ex.getMessage() + " (SQLState=" + ex.getSQLState() + ", ErrorCode=" + ex.getErrorCode() + ")";
-  }
-  static String quoteString(String input) {
-    return "'" + input + "'";
-  }
-  static String quoteChar(char c) {
-    return "'" + c + "'";
-  }
-
-  /**
-   * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
-   * Select ... For Update to sequence operations properly.  In practice that means when running
-   * with Derby database.  See more notes at class level.
-   */
-  protected void lockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.lock();
-    }
-  }
-  protected void unlockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.unlock();
-    }
-  }
   @Override
   @RetrySemantics.Idempotent

Review Comment:
   should we keep this, as it's also present in interface



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418854372


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnLockHandler.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import org.apache.hadoop.hive.common.classification.RetrySemantics;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.hadoop.hive.metastore.api.LockResponse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.ShowLocksRequest;
+import org.apache.hadoop.hive.metastore.api.ShowLocksResponse;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.api.TxnOpenException;
+import org.apache.hadoop.hive.metastore.api.UnlockRequest;
+import org.apache.hadoop.hive.metastore.txn.retryhandling.SqlRetry;
+import org.springframework.transaction.annotation.Transactional;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnStore.POOL_TX;
+
+public interface TxnLockHandler {

Review Comment:
   should we rename to `TxnLockManager`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418906994


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/commands/AddWriteIdsToMinHistoryCommand.java:
##########
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.commands;

Review Comment:
   i would remove impl package and move commands/queries/functions under jdbc package



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419021577


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +406,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table

Review Comment:
   ok, my browser is crashing on that PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419090624


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));

Review Comment:
   too long



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418812037


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##########
@@ -546,61 +518,28 @@ public long findMinOpenTxnIdForCleaner() throws MetaException {
   @RetrySemantics.Idempotent
   @Deprecated
   public long findMinTxnIdSeenOpen() {
-    if (!useMinHistoryLevel || useMinHistoryWriteId) {
+    if (!TxnHandlingFeatures.useMinHistoryLevel() || TxnHandlingFeatures.useMinHistoryWriteId()) {

Review Comment:
   ````
   TxnHandler.confVars
       .useMinHistoryLevel()
       .setUseMinHistoryLevel(boolean toggle)
       .useMinHistoryWriteId()
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419163363


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java:
##########
@@ -712,6 +782,7 @@ Set<CompactionInfo> findPotentialCompactions(int abortedThreshold, long abortedT
    * WriteSet tracking is used to ensure proper transaction isolation.  This method deletes the
    * transaction metadata once it becomes unnecessary.
    */
+  @Transactional(POOL_TX)

Review Comment:
   let's switch to COMP_POOL for APIs used only by Compaction processes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422544833


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422545472


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));

Review Comment:
   done



##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1696,41 +639,7 @@ private boolean isUpdateOrDelete(Statement stmt, String conflictSQLSuffix) throw
    */
   @RetrySemantics.ReadOnly

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422558873


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/retryhandling/SqlRetryException.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.retryhandling;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1815211360

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [18 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [167 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1398821824


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ConditionalCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import org.apache.hadoop.hive.metastore.DatabaseProduct;
+
+/**
+ * {@link ParameterizedCommand}. {@link ParameterizedBatchCommand}, and {@link InClauseBatchCommand} implementations can also
+ * implement this interface, marking that the execution is conditial, and the command wants to get notified about 
+ * execution errors. Can be used to implement commands depending on some feature flag(s).  
+ */
+public interface ConditionalCommand {
+
+  /**
+   * Indicates if the command should be executed or not
+   * @param databaseProduct
+   * @return Returns true if the command can be executed, false otherwise.
+   */
+  boolean shouldUse(DatabaseProduct databaseProduct);

Review Comment:
   I know grammar, but this is a command, that should run or not right? in this case "use" as an active verb is misleading, should be "shouldBeUsed" or rather "shouldRun", because this command should be used / this command should run



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #4566: HIVE-27481: TxnHandler cleanup

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1679007275

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![C](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C-16px.png 'C')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [8 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [124 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #4566: HIVE-27481: TxnHandler cleanup

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1673853868

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![C](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C-16px.png 'C')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [9 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [121 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1410641848


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+public class StackThreadLocal<T> {
+
+  private final ThreadLocal<ContextNode<T>> threadLocal = new ThreadLocal<>();

Review Comment:
   Good Idea, replaced my custom implementation with a Stack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418918869


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/MultiDataSourceJdbcResource.java:
##########
@@ -46,19 +57,23 @@ public class MultiDataSourceJdbcResource {
 
   private static final Logger LOG = LoggerFactory.getLogger(MultiDataSourceJdbcResource.class);
 
-  private final ThreadLocal<ContextNode<String>> threadLocal = new ThreadLocal<>();
+  private final StackThreadLocal<String> threadLocal = new StackThreadLocal<>();

Review Comment:
   should we inline StackThreadLocal impl here? it's not used anywhere else, isn't it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419187758


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/retryhandling/SqlRetryException.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.retryhandling;

Review Comment:
   maybe simply txn.retry



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419082265


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {

Review Comment:
   please add extra tab



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418837129


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          } else if (!context.isCompleted()) {
+            jdbcResource.getTransactionManager().commit(context);
+          }
           return result;
+        } catch (ProgrammaticRollbackException e) {
+          if (context != null && !context.isCompleted()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          }          
+          return e.getResult();
         } catch (Exception e) {
-          if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
-              Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
-            throw e;
-          }
           if (context != null) {
-            if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {
+            if (transactional.noRollbackFor().length > 0 || transactional.noRollbackForClassName().length > 0) {
+              if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
+                  Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
+                jdbcResource.getTransactionManager().commit(context);
+              } else {
+                jdbcResource.getTransactionManager().rollback(context);
+              }
+            } else if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {

Review Comment:
   The previous is !**no**!RollbackFor() and !**no**!RollbackForClassName()



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418834695


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -108,23 +110,36 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
         TransactionContext context = null;
         try {
           jdbcResource.bindDataSource(transactional);
-          context = jdbcResource.getTransactionManager().getTransaction(transactional.propagation().value());
+          context = jdbcResource.getTransactionManager().getNewTransaction(transactional.propagation().value());
           Object result = toCall.execute();
           LOG.debug("Successfull method invocation within transactional context: {}, going to commit.", callerId);
-          jdbcResource.getTransactionManager().commit(context);
+          if (context.isRollbackOnly()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          } else if (!context.isCompleted()) {
+            jdbcResource.getTransactionManager().commit(context);
+          }
           return result;
+        } catch (ProgrammaticRollbackException e) {
+          if (context != null && !context.isCompleted()) {
+            jdbcResource.getTransactionManager().rollback(context);
+          }          
+          return e.getResult();
         } catch (Exception e) {
-          if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
-              Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
-            throw e;
-          }
           if (context != null) {
-            if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {
+            if (transactional.noRollbackFor().length > 0 || transactional.noRollbackForClassName().length > 0) {
+              if (Arrays.stream(transactional.noRollbackFor()).anyMatch(ex -> ex.isInstance(e)) ||
+                  Arrays.stream(transactional.noRollbackForClassName()).anyMatch(exName -> exName.equals(e.getClass().getName()))) {
+                jdbcResource.getTransactionManager().commit(context);
+              } else {
+                jdbcResource.getTransactionManager().rollback(context);
+              }
+            } else if (transactional.rollbackFor().length > 0 || transactional.rollbackForClassName().length > 0) {

Review Comment:
   !! it's the same if predicate as above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419107740


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -248,199 +182,141 @@
  * the metstore call stack should have logic not to retry.  There are {@link RetrySemantics}
  * annotations to document the behavior.
  */
+@SuppressWarnings("SqlSourceToSinkFlow")
 @InterfaceAudience.Private
 @InterfaceStability.Evolving
-abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+public abstract class TxnHandler implements TxnStore, TxnStore.MutexAPI {
+
   
-  private static final String TXN_TMP_STATE = "_";
-  private static final String DEFAULT_POOL_NAME = "default";
+  public static class TxnHandlingFeatures {    

Review Comment:
   please rename to ConfVars. 
   ````
   public final ConfVars confVars;
   
   confVars = new ConfVars(table->checkIfTableIsUsable(table), conf)
   
   public static final ConfVars {
     private boolean useMinHistoryLevel;
     private boolean useMinHistoryWriteId;
     
     public ConfVars(Predicate<String> tableCheck, Conf conf){
       useMinHistoryLevel = tableCheck.test("MIN_HISTORY_WRITE_ID", 
     MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.TXN_USE_MIN_HISTORY_WRITE_ID))
     }
   }
   ````
   
   getConfVars().useMinHistoryWriteId
   getConfVars().useMinHistoryLevel



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418855723


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnLockHandlerImpl.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.hive.metastore.api.LockRequest;
+import org.apache.hadoop.hive.metastore.api.LockResponse;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.ShowLocksRequest;
+import org.apache.hadoop.hive.metastore.api.ShowLocksResponse;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.api.TxnOpenException;
+import org.apache.hadoop.hive.metastore.api.UnlockRequest;
+import org.apache.hadoop.hive.metastore.txn.entities.LockInfo;
+import org.apache.hadoop.hive.metastore.txn.impl.functions.CheckLockFunction;
+import org.apache.hadoop.hive.metastore.txn.impl.functions.EnqueueLockFunction;
+import org.apache.hadoop.hive.metastore.txn.impl.queries.GetLocksByLockId;
+import org.apache.hadoop.hive.metastore.txn.impl.queries.ShowLocksHandler;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.ProgrammaticRollbackException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.sql.Types;
+import java.util.List;
+
+import static org.apache.hadoop.hive.metastore.txn.entities.LockInfo.LOCK_WAITING;
+
+public class TxnLockHandlerImpl implements TxnLockHandler {

Review Comment:
   DefaultTxnLockManager



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418901387


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/LockHandleImpl.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl;
+
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionContext;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.dao.DataAccessException;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.util.concurrent.Semaphore;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnStore.POOL_MUTEX;
+
+public final class LockHandleImpl implements TxnStore.MutexAPI.LockHandle {

Review Comment:
   could we nest LockHandleImpl inside TxnStoreMutex(HiveMutex)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422529887


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {
+    int threshold = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD);
+    MetricsInfo metrics = jdbcResource.execute(MetricsInfoHandler.INSTANCE);
+    Set<String> resourceNames = jdbcResource.execute(new TablesWithAbortedTxnsHandler(threshold));
+    metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
+    metrics.setTablesWithXAbortedTxns(resourceNames);    
+    return metrics;
   }
 
-  /**
-   * Heartbeats on the lock table.  This commits, so do not enter it with any state.
-   * Should not be called on a lock that belongs to transaction.
-   */
-  private void heartbeatLock(Connection dbConn, long extLockId)
-    throws NoSuchLockException, SQLException, MetaException {
-    // If the lock id is 0, then there are no locks in this heartbeat
-    if (extLockId == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String updateHeartbeatQuery = "UPDATE \"HIVE_LOCKS\" SET \"HL_LAST_HEARTBEAT\" = " +
-          getEpochFn(dbProduct) + " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute update <{}>", updateHeartbeatQuery);
-      int rc = stmt.executeUpdate(updateHeartbeatQuery);
-      if (rc < 1) {
-        LOG.error("Failure to update last heartbeat for extLockId={}.", extLockId);
-        dbConn.rollback();
-        throw new NoSuchLockException("No such lock: " + JavaUtils.lockIdToString(extLockId));
-      }
-      LOG.debug("Successfully heartbeated for extLockId={}", extLockId);
-      dbConn.commit();
-    }
-  }
 
-  // Heartbeats on the txn table.  This commits, so do not enter it with any state
-  private void heartbeatTxn(Connection dbConn, long txnid)
-    throws NoSuchTxnException, TxnAbortedException, SQLException, MetaException {
-    // If the txnid is 0, then there are no transactions in this heartbeat
-    if (txnid == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String s = "UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-          " WHERE \"TXN_ID\" = " + txnid + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-      LOG.debug("Going to execute update <{}>", s);
-      int rc = stmt.executeUpdate(s);
-      if (rc < 1) {
-        ensureValidTxn(dbConn, txnid, stmt); // This should now throw some useful exception.
-        LOG.error("Can neither heartbeat txn (txnId={}) nor confirm it as invalid.", txnid);
-        dbConn.rollback();
-        throw new NoSuchTxnException("No such txn: " + txnid);
-      }
-      LOG.debug("Successfully heartbeated for txnId={}", txnid);
-      dbConn.commit();
-    }
+  private static void shouldNeverHappen(long txnid) {
+    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
   }
 
   /**
-   * Returns the state of the transaction if it's able to determine it. Some cases where it cannot:
-   * 1. txnid was Aborted/Committed and then GC'd (compacted)
-   * 2. txnid was committed but it didn't modify anything (nothing in COMPLETED_TXN_COMPONENTS)
+   * Retry-by-caller note:
+   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
+   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
    */
-  private TxnStatus findTxnState(long txnid, Statement stmt) throws SQLException, MetaException {
-    String s = "SELECT \"TXN_STATE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnid;
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        s =
-            sqlGenerator.addLimitClause(1, "1 FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_TXNID\" = "
-                + txnid);
-        LOG.debug("Going to execute query <{}>", s);
-        try (ResultSet rs2 = stmt.executeQuery(s)) {
-          if (rs2.next()) {
-            return TxnStatus.COMMITTED;
-          }
-        }
-        // could also check WRITE_SET but that seems overkill
-        return TxnStatus.UNKNOWN;
-      }
-      return TxnStatus.fromString(rs.getString(1));
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void addDynamicPartitions(AddDynamicPartitions rqst) throws NoSuchTxnException,  TxnAbortedException, MetaException {
+    TxnType txnType = jdbcResource.execute(new GetOpenTxnTypeAndLockHandler(sqlGenerator, rqst.getTxnid()));
+    if (txnType == null) {
+      //ensures txn is still there and in expected state
+      new EnsureValidTxnFunction(rqst.getTxnid()).execute(jdbcResource);
+      shouldNeverHappen(rqst.getTxnid());
+    }
+    jdbcResource.execute(new InsertTxnComponentsCommand(rqst));
+    jdbcResource.getJdbcTemplate().update("DELETE FROM \"TXN_COMPONENTS\" " +
+            "WHERE \"TC_TXNID\" = :txnId AND \"TC_DATABASE\" = :dbName AND \"TC_TABLE\" = :tableName AND \"TC_PARTITION\" IS NULL",
+        new MapSqlParameterSource()
+            .addValue("txnId", rqst.getTxnid())
+            .addValue("dbName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getDbname()))
+            .addValue("tableName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getTablename())));
   }
 
   /**
-   * Checks if all the txns in the list are in open state and not read-only.
-   * @param txnIds list of txns to be evaluated for open state/read-only status
-   * @param stmt db statement
-   * @return If all the txns in open state and not read-only, then return true else false
+   * Clean up corresponding records in metastore tables when corresponding object is dropped,
+   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
+   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
+   * operations.
+   * <p>
+   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
    */
-  private boolean isTxnsOpenAndNotReadOnly(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Get the count of txns from the given list that are in open state and not read-only.
-    // If the returned count is same as the input number of txns, then all txns are in open state and not read-only.
-    prefix.append("SELECT COUNT(*) FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN
-        + " AND \"TXN_TYPE\" != " + TxnType.READ_ONLY.getValue() + " AND ");
-
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-
-    long count = 0;
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        if (rs.next()) {
-          count += rs.getLong(1);
-        }
-      }
-    }
-    return count == txnIds.size();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), keepTxnToWriteIdMetaData, null)
+        .execute(jdbcResource);
   }
 
-  /**
-   * Get txns from the list that are either aborted or read-only.
-   * @param txnIds list of txns to be evaluated for aborted state/read-only status
-   * @param stmt db statement
-   */
-  private String getAbortedAndReadOnlyTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are either aborted or read-only.
-    prefix.append("SELECT \"TXN_ID\", \"TXN_STATE\", \"TXN_TYPE\" FROM \"TXNS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          TxnStatus txnState = TxnStatus.fromString(rs.getString(2));
-          TxnType txnType = TxnType.findByValue(rs.getInt(3));
-
-          if (txnState != TxnStatus.OPEN) {
-            txnInfo.append("{").append(txnId).append(",").append(txnState).append("}");
-          } else if (txnType == TxnType.READ_ONLY) {
-            txnInfo.append("{").append(txnId).append(",read-only}");
-          }
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), false, txnId)
+        .execute(jdbcResource);
   }
-
+  
   /**
-   * Get txns from the list that are committed.
-   * @param txnIds list of txns to be evaluated for committed state
-   * @param stmt db statement
+   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
    */
-  private String getCommittedTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are committed.
-    prefix.append("SELECT \"CTC_TXNID\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"CTC_TXNID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          txnInfo.append("{").append(txnId).append(",c}");
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
+      String newCatName, String newDbName, String newTabName, String newPartName)
+      throws MetaException {
+    new OnRenameFunction(oldCatName, oldDbName, oldTabName, oldPartName, newCatName, newDbName, newTabName, newPartName).execute(jdbcResource);
   }
-
   /**
-   * Used to raise an informative error when the caller expected a txn in a particular TxnStatus
-   * but found it in some other status
+   * For testing only, do not use.
    */
-  private static void raiseTxnUnexpectedState(TxnStatus actualStatus, long txnid)
-    throws NoSuchTxnException, TxnAbortedException {
-    switch (actualStatus) {
-      case ABORTED:
-        throw new TxnAbortedException("Transaction " + JavaUtils.txnIdToString(txnid) + " already aborted");
-      case COMMITTED:
-        throw new NoSuchTxnException("Transaction " + JavaUtils.txnIdToString(txnid) + " is already committed.");
-      case UNKNOWN:
-        throw new NoSuchTxnException("No such transaction " + JavaUtils.txnIdToString(txnid));
-      case OPEN:
-        throw new NoSuchTxnException(JavaUtils.txnIdToString(txnid) + " is " + TxnStatus.OPEN);
-      default:
-        throw new IllegalArgumentException("Unknown TxnStatus " + actualStatus);
-    }
+  @VisibleForTesting
+  public int numLocksInLockTable() {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1422533468


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {
+    int threshold = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD);
+    MetricsInfo metrics = jdbcResource.execute(MetricsInfoHandler.INSTANCE);
+    Set<String> resourceNames = jdbcResource.execute(new TablesWithAbortedTxnsHandler(threshold));
+    metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
+    metrics.setTablesWithXAbortedTxns(resourceNames);    
+    return metrics;
   }
 
-  /**
-   * Heartbeats on the lock table.  This commits, so do not enter it with any state.
-   * Should not be called on a lock that belongs to transaction.
-   */
-  private void heartbeatLock(Connection dbConn, long extLockId)
-    throws NoSuchLockException, SQLException, MetaException {
-    // If the lock id is 0, then there are no locks in this heartbeat
-    if (extLockId == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String updateHeartbeatQuery = "UPDATE \"HIVE_LOCKS\" SET \"HL_LAST_HEARTBEAT\" = " +
-          getEpochFn(dbProduct) + " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute update <{}>", updateHeartbeatQuery);
-      int rc = stmt.executeUpdate(updateHeartbeatQuery);
-      if (rc < 1) {
-        LOG.error("Failure to update last heartbeat for extLockId={}.", extLockId);
-        dbConn.rollback();
-        throw new NoSuchLockException("No such lock: " + JavaUtils.lockIdToString(extLockId));
-      }
-      LOG.debug("Successfully heartbeated for extLockId={}", extLockId);
-      dbConn.commit();
-    }
-  }
 
-  // Heartbeats on the txn table.  This commits, so do not enter it with any state
-  private void heartbeatTxn(Connection dbConn, long txnid)
-    throws NoSuchTxnException, TxnAbortedException, SQLException, MetaException {
-    // If the txnid is 0, then there are no transactions in this heartbeat
-    if (txnid == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String s = "UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-          " WHERE \"TXN_ID\" = " + txnid + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-      LOG.debug("Going to execute update <{}>", s);
-      int rc = stmt.executeUpdate(s);
-      if (rc < 1) {
-        ensureValidTxn(dbConn, txnid, stmt); // This should now throw some useful exception.
-        LOG.error("Can neither heartbeat txn (txnId={}) nor confirm it as invalid.", txnid);
-        dbConn.rollback();
-        throw new NoSuchTxnException("No such txn: " + txnid);
-      }
-      LOG.debug("Successfully heartbeated for txnId={}", txnid);
-      dbConn.commit();
-    }
+  private static void shouldNeverHappen(long txnid) {
+    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
   }
 
   /**
-   * Returns the state of the transaction if it's able to determine it. Some cases where it cannot:
-   * 1. txnid was Aborted/Committed and then GC'd (compacted)
-   * 2. txnid was committed but it didn't modify anything (nothing in COMPLETED_TXN_COMPONENTS)
+   * Retry-by-caller note:
+   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
+   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
    */
-  private TxnStatus findTxnState(long txnid, Statement stmt) throws SQLException, MetaException {
-    String s = "SELECT \"TXN_STATE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnid;
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        s =
-            sqlGenerator.addLimitClause(1, "1 FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_TXNID\" = "
-                + txnid);
-        LOG.debug("Going to execute query <{}>", s);
-        try (ResultSet rs2 = stmt.executeQuery(s)) {
-          if (rs2.next()) {
-            return TxnStatus.COMMITTED;
-          }
-        }
-        // could also check WRITE_SET but that seems overkill
-        return TxnStatus.UNKNOWN;
-      }
-      return TxnStatus.fromString(rs.getString(1));
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void addDynamicPartitions(AddDynamicPartitions rqst) throws NoSuchTxnException,  TxnAbortedException, MetaException {
+    TxnType txnType = jdbcResource.execute(new GetOpenTxnTypeAndLockHandler(sqlGenerator, rqst.getTxnid()));
+    if (txnType == null) {
+      //ensures txn is still there and in expected state
+      new EnsureValidTxnFunction(rqst.getTxnid()).execute(jdbcResource);
+      shouldNeverHappen(rqst.getTxnid());
+    }
+    jdbcResource.execute(new InsertTxnComponentsCommand(rqst));
+    jdbcResource.getJdbcTemplate().update("DELETE FROM \"TXN_COMPONENTS\" " +
+            "WHERE \"TC_TXNID\" = :txnId AND \"TC_DATABASE\" = :dbName AND \"TC_TABLE\" = :tableName AND \"TC_PARTITION\" IS NULL",
+        new MapSqlParameterSource()
+            .addValue("txnId", rqst.getTxnid())
+            .addValue("dbName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getDbname()))
+            .addValue("tableName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getTablename())));
   }
 
   /**
-   * Checks if all the txns in the list are in open state and not read-only.
-   * @param txnIds list of txns to be evaluated for open state/read-only status
-   * @param stmt db statement
-   * @return If all the txns in open state and not read-only, then return true else false
+   * Clean up corresponding records in metastore tables when corresponding object is dropped,
+   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
+   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
+   * operations.
+   * <p>
+   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
    */
-  private boolean isTxnsOpenAndNotReadOnly(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Get the count of txns from the given list that are in open state and not read-only.
-    // If the returned count is same as the input number of txns, then all txns are in open state and not read-only.
-    prefix.append("SELECT COUNT(*) FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN
-        + " AND \"TXN_TYPE\" != " + TxnType.READ_ONLY.getValue() + " AND ");
-
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-
-    long count = 0;
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        if (rs.next()) {
-          count += rs.getLong(1);
-        }
-      }
-    }
-    return count == txnIds.size();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), keepTxnToWriteIdMetaData, null)
+        .execute(jdbcResource);
   }
 
-  /**
-   * Get txns from the list that are either aborted or read-only.
-   * @param txnIds list of txns to be evaluated for aborted state/read-only status
-   * @param stmt db statement
-   */
-  private String getAbortedAndReadOnlyTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are either aborted or read-only.
-    prefix.append("SELECT \"TXN_ID\", \"TXN_STATE\", \"TXN_TYPE\" FROM \"TXNS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          TxnStatus txnState = TxnStatus.fromString(rs.getString(2));
-          TxnType txnType = TxnType.findByValue(rs.getInt(3));
-
-          if (txnState != TxnStatus.OPEN) {
-            txnInfo.append("{").append(txnId).append(",").append(txnState).append("}");
-          } else if (txnType == TxnType.READ_ONLY) {
-            txnInfo.append("{").append(txnId).append(",read-only}");
-          }
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), false, txnId)
+        .execute(jdbcResource);
   }
-
+  
   /**
-   * Get txns from the list that are committed.
-   * @param txnIds list of txns to be evaluated for committed state
-   * @param stmt db statement
+   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
    */
-  private String getCommittedTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are committed.
-    prefix.append("SELECT \"CTC_TXNID\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"CTC_TXNID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          txnInfo.append("{").append(txnId).append(",c}");
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
+      String newCatName, String newDbName, String newTabName, String newPartName)
+      throws MetaException {
+    new OnRenameFunction(oldCatName, oldDbName, oldTabName, oldPartName, newCatName, newDbName, newTabName, newPartName).execute(jdbcResource);
   }
-
   /**
-   * Used to raise an informative error when the caller expected a txn in a particular TxnStatus
-   * but found it in some other status
+   * For testing only, do not use.
    */
-  private static void raiseTxnUnexpectedState(TxnStatus actualStatus, long txnid)
-    throws NoSuchTxnException, TxnAbortedException {
-    switch (actualStatus) {
-      case ABORTED:
-        throw new TxnAbortedException("Transaction " + JavaUtils.txnIdToString(txnid) + " already aborted");
-      case COMMITTED:
-        throw new NoSuchTxnException("Transaction " + JavaUtils.txnIdToString(txnid) + " is already committed.");
-      case UNKNOWN:
-        throw new NoSuchTxnException("No such transaction " + JavaUtils.txnIdToString(txnid));
-      case OPEN:
-        throw new NoSuchTxnException(JavaUtils.txnIdToString(txnid) + " is " + TxnStatus.OPEN);
-      default:
-        throw new IllegalArgumentException("Unknown TxnStatus " + actualStatus);
-    }
+  @VisibleForTesting
+  public int numLocksInLockTable() {
+    return Objects.requireNonNull(
+        jdbcResource.getJdbcTemplate().queryForObject("SELECT COUNT(*) FROM \"HIVE_LOCKS\"", new MapSqlParameterSource(), Integer.TYPE),
+        "This never should be null, it's just to suppress warnings");
   }
+
   /**
-   * Returns the state of the transaction with {@code txnid} or throws if {@code raiseError} is true.
+   * For testing only, do not use.

Review Comment:
   I agree, but I think this should be handled separately. Currently this is the only way to alter the `timeout` field



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425288521


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),

Review Comment:
   ok, but can we create a wrapper `lock` method without any retry for simplicity?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425292515


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ConditionalCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import org.apache.hadoop.hive.metastore.DatabaseProduct;
+
+/**
+ * {@link ParameterizedCommand}. {@link ParameterizedBatchCommand}, and {@link InClauseBatchCommand} implementations can also
+ * implement this interface, marking that the execution is conditial, and the command wants to get notified about 
+ * execution errors. Can be used to implement commands depending on some feature flag(s).  
+ */
+public interface ConditionalCommand {
+
+  /**
+   * Indicates if the command should be executed or not
+   * @param databaseProduct
+   * @return Returns true if the command can be executed, false otherwise.
+   */
+  boolean shouldBeUsed(DatabaseProduct databaseProduct);

Review Comment:
   The `ConfVars` (TxnHandlingFeatures) are containing flags like do we have the useMinHistoryLevel or useMinHistoryWriteId tables. Some commands are depending on the existence of these tables. An example: `RemoveTxnsFromMinHistoryLevelCommand`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1857773914

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [215 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 merged PR #4566:
URL: https://github.com/apache/hive/pull/4566


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1858536753

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [215 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1812974188

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [17 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [164 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425275159


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +400,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table
+    StopWatch generateTransactionWatch = new StopWatch();
+    generateTransactionWatch.start();
 
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        generateTransactionWatch.stop();
-        long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
-        TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-        if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
-          /*
-           * The commit was too slow, we can not allow this to continue (except if it is read only,
-           * since that can not cause dirty reads).
-           * When calculating the snapshot for a given transaction, we look back for possible open transactions
-           * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
-           * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
-           * because there can be other transactions running, that didn't considered this transactionId open,
-           * this could cause dirty reads.
-           */
-          LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
-          deleteInvalidOpenTransactions(dbConn, txnIds);
-          dbConn.commit();
-          /*
-           * We do not throw RetryException directly, to not circumvent the max retry limit
-           */
-          throw new SQLException("OpenTxnTimeOut exceeded", MANUAL_RETRY);
-        }
-        return new OpenTxnsResponse(txnIds);
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "openTxns(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return openTxns(rqst);
-    }
-  }
+    List<Long> txnIds = new OpenTxnsFunction(rqst, openTxnTimeOutMillis, transactionalListeners).execute(jdbcResource);
 
-  private List<Long> openTxns(Connection dbConn, OpenTxnRequest rqst)
-          throws SQLException, MetaException {
-    int numTxns = rqst.getNum_txns();
-    // Make sure the user has not requested an insane amount of txns.
-    int maxTxns = MetastoreConf.getIntVar(conf, ConfVars.TXN_MAX_OPEN_BATCH);
-    if (numTxns > maxTxns) {
-      numTxns = maxTxns;
-    }
-    List<PreparedStatement> insertPreparedStmts = null;
+    LOG.debug("Going to commit");
+    jdbcResource.getTransactionManager().getActiveTransaction().createSavepoint();
+    generateTransactionWatch.stop();
+    long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
     TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-    boolean isReplayedReplTxn = txnType == TxnType.REPL_CREATED;
-    boolean isHiveReplTxn = rqst.isSetReplPolicy() && txnType == TxnType.DEFAULT;
-    try {
-      if (isReplayedReplTxn) {
-        assert rqst.isSetReplPolicy();
-        List<Long> targetTxnIdList = getTargetTxnIdList(rqst.getReplPolicy(), rqst.getReplSrcTxnIds(), dbConn);
-
-        if (!targetTxnIdList.isEmpty()) {
-          if (targetTxnIdList.size() != rqst.getReplSrcTxnIds().size()) {
-            LOG.warn("target txn id number {} is not matching with source txn id number {}",
-                targetTxnIdList, rqst.getReplSrcTxnIds());
-          }
-          LOG.info("Target transactions {} are present for repl policy : {} and Source transaction id : {}",
-              targetTxnIdList.toString(), rqst.getReplPolicy(), rqst.getReplSrcTxnIds().toString());
-          return targetTxnIdList;
-        }
-      }
-
-      long minOpenTxnId = 0;
-      if (useMinHistoryLevel) {
-        minOpenTxnId = getMinOpenTxnIdWaterMark(dbConn);
-      }
-
-      List<Long> txnIds = new ArrayList<>(numTxns);
+    if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
       /*
-       * The getGeneratedKeys are not supported in every dbms, after executing a multi line insert.
-       * But it is supported in every used dbms for single line insert, even if the metadata says otherwise.
-       * If the getGeneratedKeys are not supported first we insert a random batchId in the TXN_META_INFO field,
-       * then the keys are selected beck with that batchid.
+       * The commit was too slow, we can not allow this to continue (except if it is read only,
+       * since that can not cause dirty reads).
+       * When calculating the snapshot for a given transaction, we look back for possible open transactions
+       * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
+       * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
+       * because there can be other transactions running, that didn't considered this transactionId open,
+       * this could cause dirty reads.
        */
-      boolean genKeySupport = dbProduct.supportsGetGeneratedKeys();
-      genKeySupport = genKeySupport || (numTxns == 1);
-
-      String insertQuery = String.format(TXNS_INSERT_QRY, getEpochFn(dbProduct), getEpochFn(dbProduct));
-      LOG.debug("Going to execute insert <{}>", insertQuery);
-      try (PreparedStatement ps = dbConn.prepareStatement(insertQuery, new String[] {"TXN_ID"})) {
-        String state = genKeySupport ? TxnStatus.OPEN.getSqlConst() : TXN_TMP_STATE;
-        if (numTxns == 1) {
-          ps.setString(1, state);
-          ps.setString(2, rqst.getUser());
-          ps.setString(3, rqst.getHostname());
-          ps.setInt(4, txnType.getValue());
-          txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, false));
-        } else {
-          for (int i = 0; i < numTxns; ++i) {
-            ps.setString(1, state);
-            ps.setString(2, rqst.getUser());
-            ps.setString(3, rqst.getHostname());
-            ps.setInt(4, txnType.getValue());
-            ps.addBatch();
-
-            if ((i + 1) % maxBatchSize == 0) {
-              txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, true));
-            }
-          }
-          if (numTxns % maxBatchSize != 0) {
-            txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, true));
-          }
-        }
-      }
-
-      assert txnIds.size() == numTxns;
-
-      addTxnToMinHistoryLevel(dbConn, txnIds, minOpenTxnId);
-
-      if (isReplayedReplTxn) {
-        List<String> rowsRepl = new ArrayList<>(numTxns);
-        List<String> params = Collections.singletonList(rqst.getReplPolicy());
-        List<List<String>> paramsList = new ArrayList<>(numTxns);
-        for (int i = 0; i < numTxns; i++) {
-          rowsRepl.add("?," + rqst.getReplSrcTxnIds().get(i) + "," + txnIds.get(i));
-          paramsList.add(params);
-        }
-
-        insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                "\"REPL_TXN_MAP\" (\"RTM_REPL_POLICY\", \"RTM_SRC_TXN_ID\", \"RTM_TARGET_TXN_ID\")", rowsRepl,
-                paramsList);
-        for (PreparedStatement pst : insertPreparedStmts) {
-          pst.execute();
-        }
-      }
-
-      if (transactionalListeners != null && !isHiveReplTxn) {
-        MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                EventMessage.EventType.OPEN_TXN, new OpenTxnEvent(txnIds, txnType), dbConn, sqlGenerator);
-      }
-      return txnIds;
-    } finally {
-      if (insertPreparedStmts != null) {
-        for (PreparedStatement pst : insertPreparedStmts) {
-          pst.close();
-        }
-      }
-    }
-  }
+      LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
 
-  private List<Long> executeTxnInsertBatchAndExtractGeneratedKeys(Connection dbConn, boolean genKeySupport,
-      PreparedStatement ps, boolean batch) throws SQLException {
-    List<Long> txnIds = new ArrayList<>();
-    if (batch) {
-      ps.executeBatch();
-    } else {
-      // For slight performance advantage we do not use the executeBatch, when we only have one row
-      ps.execute();
-    }
-    if (genKeySupport) {
-      try (ResultSet generatedKeys = ps.getGeneratedKeys()) {
-        while (generatedKeys.next()) {
-          txnIds.add(generatedKeys.getLong(1));
-        }
-      }
-    } else {
-      try (PreparedStatement pstmt =
-          dbConn.prepareStatement("SELECT \"TXN_ID\" FROM \"TXNS\" WHERE \"TXN_STATE\" = ?")) {
-        pstmt.setString(1, TXN_TMP_STATE);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          while (rs.next()) {
-            txnIds.add(rs.getLong(1));
-          }
+      if (!txnIds.isEmpty()) {
+        try {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1854158870

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566) **Quality Gate passed**  
   The SonarCloud Quality Gate passed, but some issues were introduced.
   
   [223 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&inNewCodePeriod=true)  
   No data about Coverage  
   No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425277755


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),
+          rqst.isExclusiveCTAS());
+    } catch (NoSuchLockException e) {
+      // This should never happen, as we just added the lock id
+      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());      
     }
   }
-
+  
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @param conn database connection
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
+   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
+   * <p>
+   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
+   * in practice more often)
+   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
+   * <p>
+   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
+   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
+   * <p>
+   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
+   * (make sure some lock we are checking against doesn't move from W to A in another txn)
+   * but this method can heartbeat in separate txn at READ_COMMITTED.
+   * <p>
+   * Retry-by-caller note:
+   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
    */
-  protected long getDbTime(Connection conn) throws MetaException {
-    Statement stmt = null;
-    try {
-      stmt = conn.createStatement();
-      String s = dbProduct.getDBTime();
-
-      LOG.debug("Going to execute query <{}>", s);
-      ResultSet rs = stmt.executeQuery(s);
-      if (!rs.next()) throw new MetaException("No results from date query");
-      return rs.getTimestamp(1).getTime();
-    } catch (SQLException e) {
-      String msg = "Unable to determine current time: " + e.getMessage();
-      LOG.error(msg);
-      throw new MetaException(msg);
-    } finally {
-      closeStmt(stmt);
+  @Override
+  @RetrySemantics.SafeToRetry
+  public LockResponse checkLock(CheckLockRequest rqst)
+      throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
+    long extLockId = rqst.getLockid();
+    // Heartbeat on the lockid first, to assure that our lock is still valid.
+    // Then look up the lock info (hopefully in the cache).  If these locks
+    // are associated with a transaction then heartbeat on that as well.
+    List<LockInfo> lockInfos = jdbcResource.execute(new GetLocksByLockId(extLockId, 1, sqlGenerator));
+    if (CollectionUtils.isEmpty(lockInfos)) {
+      throw new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId));
+    }
+    LockInfo lockInfo = lockInfos.get(0);
+    if (lockInfo.getTxnId() > 0) {
+      new HeartbeatTxnFunction(lockInfo.getTxnId()).execute(jdbcResource);
+    } else {
+      new HeartBeatLockFunction(rqst.getLockid()).execute(jdbcResource);
     }
+    return txnLockManager.checkLock(extLockId, lockInfo.getTxnId(), false, false);
   }
 
   /**
-   * Determine the current time, using the RDBMS as a source of truth
-   * @return current time in milliseconds
-   * @throws org.apache.hadoop.hive.metastore.api.MetaException if the time cannot be determined
+   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
+   * be heartbeated, committed, etc.  no need for client to track individual locks.
+   * When removing locks not associated with txn this potentially conflicts with
+   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
+   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
    */
-  protected Timestamp getDbTime() throws MetaException {
-    return jdbcResource.getJdbcTemplate().queryForObject(
-        dbProduct.getDBTime(), 
-        new MapSqlParameterSource(),
-        (ResultSet rs, int rowNum) -> rs.getTimestamp(1));
+  @RetrySemantics.Idempotent
+  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
+    txnLockManager.unlock(rqst);
   }
-  
 
-  protected String isWithinCheckInterval(String expr, long interval) throws MetaException {
-    return dbProduct.isWithinCheckInterval(expr, interval);
+  @RetrySemantics.ReadOnly
+  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
+    return txnLockManager.showLocks(rqst);
   }
 
-  private void determineDatabaseProduct(Connection conn) {
-    try {
-      String s = conn.getMetaData().getDatabaseProductName();
-      dbProduct = DatabaseProduct.determineDatabaseProduct(s, conf);
-      if (dbProduct.isUNDEFINED()) {
-        String msg = "Unrecognized database product name <" + s + ">";
-        LOG.error(msg);
-        throw new IllegalStateException(msg);
-      }
-    } catch (SQLException e) {
-      String msg = "Unable to get database product name";
-      LOG.error(msg, e);
-      throw new IllegalStateException(msg, e);
-    }
+  /**
+   * {@code ids} should only have txnid or lockid but not both, ideally.
+   * Currently DBTxnManager.heartbeat() enforces this.
+   */
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void heartbeat(HeartbeatRequest ids)
+    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
+    new HeartbeatTxnFunction(ids.getTxnid()).execute(jdbcResource);
+    new HeartBeatLockFunction(ids.getLockid()).execute(jdbcResource);
   }
-
-  private static class LockInfo {
-    private final long extLockId;
-    private final long intLockId;
-    //0 means there is no transaction, i.e. it a select statement which is not part of
-    //explicit transaction or a IUD statement that is not writing to ACID table
-    private final long txnId;
-    private final String db;
-    private final String table;
-    private final String partition;
-    private final LockState state;
-    private final LockType type;
-
-    // Assumes the result set is set to a valid row
-    LockInfo(ResultSet rs) throws SQLException, MetaException {
-      extLockId = rs.getLong("HL_LOCK_EXT_ID"); // can't be null
-      intLockId = rs.getLong("HL_LOCK_INT_ID"); // can't be null
-      db = rs.getString("HL_DB"); // can't be null
-      String t = rs.getString("HL_TABLE");
-      table = (rs.wasNull() ? null : t);
-      String p = rs.getString("HL_PARTITION");
-      partition = (rs.wasNull() ? null : p);
-      switch (rs.getString("HL_LOCK_STATE").charAt(0)) {
-        case LOCK_WAITING: state = LockState.WAITING; break;
-        case LOCK_ACQUIRED: state = LockState.ACQUIRED; break;
-        default:
-          throw new MetaException("Unknown lock state " + rs.getString("HL_LOCK_STATE").charAt(0));
-      }
-      char lockChar = rs.getString("HL_LOCK_TYPE").charAt(0);
-      type = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-              .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-      txnId = rs.getLong("HL_TXNID"); //returns 0 if value is NULL
-    }
-
-    LockInfo(ShowLocksResponseElement e) {
-      extLockId = e.getLockid();
-      intLockId = e.getLockIdInternal();
-      txnId = e.getTxnid();
-      db = e.getDbname();
-      table = e.getTablename();
-      partition = e.getPartname();
-      state = e.getState();
-      type = e.getType();
-    }
-
-    public boolean equals(Object other) {
-      if (!(other instanceof LockInfo)) return false;
-      LockInfo o = (LockInfo)other;
-      // Lock ids are unique across the system.
-      return extLockId == o.extLockId && intLockId == o.intLockId;
-    }
-
-    @Override
-    public String toString() {
-      return JavaUtils.lockIdToString(extLockId) + " intLockId:" +
-        intLockId + " " + JavaUtils.txnIdToString(txnId)
-        + " db:" + db + " table:" + table + " partition:" +
-        partition + " state:" + (state == null ? "null" : state.toString())
-        + " type:" + (type == null ? "null" : type.toString());
-    }
-    private boolean isDbLock() {
-      return db != null && table == null && partition == null;
-    }
-    private boolean isTableLock() {
-      return db != null && table != null && partition == null;
-    }
-    private boolean isPartitionLock() {
-      return !(isDbLock() || isTableLock());
-    }
+  
+  @Override
+  @RetrySemantics.SafeToRetry
+  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst) throws MetaException {
+    return new HeartbeatTxnRangeFunction(rqst).execute(jdbcResource);
   }
 
-  private static class LockInfoComparator implements Comparator<LockInfo>, Serializable {
-    private LockTypeComparator lockTypeComparator = new LockTypeComparator();
-
-    public boolean equals(Object other) {
-      return this == other;
-    }
-
-    public int compare(LockInfo info1, LockInfo info2) {
-      // We sort by state (acquired vs waiting) and then by LockType, then by id
-      if (info1.state == LockState.ACQUIRED &&
-        info2.state != LockState .ACQUIRED) {
-        return -1;
-      }
-      if (info1.state != LockState.ACQUIRED &&
-        info2.state == LockState .ACQUIRED) {
-        return 1;
-      }
-
-      int sortByType = lockTypeComparator.compare(info1.type, info2.type);
-      if(sortByType != 0) {
-        return sortByType;
-      }
-      if (info1.extLockId < info2.extLockId) {
-        return -1;
-      } else if (info1.extLockId > info2.extLockId) {
-        return 1;
-      } else {
-        if (info1.intLockId < info2.intLockId) {
-          return -1;
-        } else if (info1.intLockId > info2.intLockId) {
-          return 1;
-        } else {
-          return 0;
-        }
-      }
-    }
+  @Override
+  @RetrySemantics.ReadOnly
+  public long getTxnIdForWriteId(String dbName, String tblName, long writeId) throws MetaException {
+    return jdbcResource.execute(new TxnIdForWriteIdHandler(writeId, dbName, tblName));
   }
 
-  private enum LockAction {ACQUIRE, WAIT, KEEP_LOOKING}
-
-  // A jump table to figure out whether to wait, acquire,
-  // or keep looking .  Since
-  // java doesn't have function pointers (grumble grumble) we store a
-  // character that we'll use to determine which function to call.
-  // The table maps the lock type of the lock we are looking to acquire to
-  // the lock type of the lock we are checking to the lock state of the lock
-  // we are checking to the desired action.
-  private static Map<LockType, Map<LockType, Map<LockState, LockAction>>> jumpTable;
-
-  private int abortTxns(Connection dbConn, List<Long> txnids,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg) throws SQLException, MetaException {
-    return abortTxns(dbConn, txnids, false, skipCount, isReplReplayed, txnErrorMsg);
+  @Override
+  @RetrySemantics.Idempotent
+  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
+    return new CompactFunction(rqst, openTxnTimeOutMillis, getMutexAPI()).execute(jdbcResource);
   }
-  /**
-   * TODO: expose this as an operation to client.  Useful for streaming API to abort all remaining
-   * transactions in a batch on IOExceptions.
-   * Caller must rollback the transaction if not all transactions were aborted since this will not
-   * attempt to delete associated locks in this case.
-   *
-   * @param dbConn An active connection
-   * @param txnids list of transactions to abort
-   * @param checkHeartbeat value used by {@link #performTimeOuts()} to ensure this doesn't Abort txn which were
-   *                      heartbeated after #performTimeOuts() select and this operation.
-   * @param skipCount If true, the method always returns 0, otherwise returns the number of actually aborted txns
-   * @return 0 if skipCount is true, the number of aborted transactions otherwise
-   * @throws SQLException
-   */
-  private int abortTxns(Connection dbConn, List<Long> txnids, boolean checkHeartbeat,
-                        boolean skipCount, boolean isReplReplayed, TxnErrorMsg txnErrorMsg)
-      throws SQLException, MetaException {
-    Statement stmt = null;
-    if (txnids.isEmpty()) {
-      return 0;
-    }
-    Collections.sort(txnids);
-    LOG.debug("Aborting {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-    removeTxnsFromMinHistoryLevel(dbConn, txnids);
-    removeWriteIdsFromMinHistory(dbConn, txnids);
-    try {
-      stmt = dbConn.createStatement();
-      //This is an update statement, thus at any Isolation level will take Write locks so will block
-      //all other ops using S4U on TXNS row.
-      List<String> queries = new ArrayList<>();
-      StringBuilder prefix = new StringBuilder();
-      StringBuilder suffix = new StringBuilder();
-
-      // add update txns queries to query list
-      prefix.append("UPDATE \"TXNS\" SET \"TXN_STATE\" = ").append(TxnStatus.ABORTED)
-              .append(" , \"TXN_META_INFO\" = ").append(txnErrorMsg.toSqlString())
-              .append(" WHERE \"TXN_STATE\" = ").append(TxnStatus.OPEN).append(" AND ");
-      if (checkHeartbeat) {
-        suffix.append(" AND \"TXN_LAST_HEARTBEAT\" < ")
-                .append(getEpochFn(dbProduct)).append("-").append(timeout);
-      }
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"TXN_ID\"", true, false);
-      int numUpdateQueries = queries.size();
 
-      // add delete hive locks queries to query list
-      prefix.setLength(0);
-      suffix.setLength(0);
-      prefix.append("DELETE FROM \"HIVE_LOCKS\" WHERE ");
-      TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"HL_TXNID\"", false, false);
-
-      //If this abort is for REPL_CREATED TXN initiated outside the replication flow, then clean the corresponding entry
-      //from REPL_TXN_MAP and mark that database as replication incompatible.
-      if (!isReplReplayed) {
-        for (String database : getDbNamesForReplayedTxns(dbConn, txnids)) {
-          markDbAsReplIncompatible(dbConn, database);
-        }
-        // Delete mapping from REPL_TXN_MAP if it exists.
-        prefix.setLength(0);
-        suffix.setLength(0);
-        prefix.append("DELETE FROM \"REPL_TXN_MAP\" WHERE ");
-        TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix, txnids, "\"RTM_TARGET_TXN_ID\"", false, false);
-      }
-
-      // execute all queries in the list in one batch
-      int numAborted = 0;
-      if (skipCount) {
-        executeQueriesInBatchNoCount(dbProduct, stmt, queries, maxBatchSize);
-      } else {
-        List<Integer> affectedRowsByQuery = executeQueriesInBatch(stmt, queries, maxBatchSize);
-        numAborted = getUpdateCount(numUpdateQueries, affectedRowsByQuery);
-      }
-
-      if (MetastoreConf.getBoolVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
-        Metrics.getOrCreateCounter(MetricsConstants.TOTAL_NUM_ABORTED_TXNS).inc(txnids.size());
-      }
-      LOG.warn("Aborted {} transaction(s) {} due to {}", txnids.size(), txnids, txnErrorMsg);
-      return numAborted;
-    } finally {
-      closeStmt(stmt);
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
+    // Put a compaction request in the queue.
+    long id = new GenerateCompactionQueueIdFunction().execute(jdbcResource);
+    jdbcResource.execute(new InsertCompactionRequestCommand(id, CompactionState.READY_FOR_CLEANING, rqst).withTxnDetails(highestWriteId, txnId));
+    return true;
   }
-
-  private int getUpdateCount(int numUpdateQueries, List<Integer> affectedRowsByQuery) {
-    return affectedRowsByQuery.stream()
-            .limit(numUpdateQueries)
-            .mapToInt(Integer::intValue)
-            .sum();
+  
+  @RetrySemantics.ReadOnly
+  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
+    return jdbcResource.execute(new ShowCompactHandler(rqst, sqlGenerator));
   }
 
-  private static boolean isValidTxn(long txnId) {
-    return txnId != 0;
-  }
   /**
-   * Lock acquisition is meant to be fair, so every lock can only block on some lock with smaller
-   * hl_lock_ext_id by only checking earlier locks.
-   *
-   * For any given SQL statement all locks required by it are grouped under single extLockId and are
-   * granted all at once or all locks wait.
-   *
-   * This is expected to run at READ_COMMITTED.
-   *
-   * If there is a concurrent commitTxn/rollbackTxn, those can only remove rows from HIVE_LOCKS.
-   * If they happen to be for the same txnid, there will be a WW conflict (in MS DB), if different txnid,
-   * checkLock() will in the worst case keep locks in Waiting state a little longer.
+   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
+   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
+   * only these two states means there are new files ready to be read.
    */
-  @RetrySemantics.SafeToRetry("See @SafeToRetry")
-  private LockResponse checkLock(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException, SQLException {
-    Statement stmt = null;
-    ResultSet rs = null;
-    LockResponse response = new LockResponse();
-    /**
-     * todo: Longer term we should pass this from client somehow - this would be an optimization;  once
-     * that is in place make sure to build and test "writeSet" below using OperationType not LockType
-     * With Static Partitions we assume that the query modifies exactly the partitions it locked.  (not entirely
-     * realistic since Update/Delete may have some predicate that filters out all records out of
-     * some partition(s), but plausible).  For DP, we acquire locks very wide (all known partitions),
-     * but for most queries only a fraction will actually be updated.  #addDynamicPartitions() tells
-     * us exactly which ones were written to.  Thus using this trick to kill a query early for
-     * DP queries may be too restrictive.
-     */
-    boolean isPartOfDynamicPartitionInsert = true;
-    try {
-      List<LockInfo> locksBeingChecked = getLocksFromLockId(dbConn, extLockId); //being acquired now
-      response.setLockid(extLockId);
-
-      //This is the set of entities that the statement represented by extLockId wants to update
-      List<LockInfo> writeSet = new ArrayList<>();
-
-      for (LockInfo info : locksBeingChecked) {
-        if(!isPartOfDynamicPartitionInsert && info.type == LockType.SHARED_WRITE) {
-          writeSet.add(info);
-        }
-      }
-      if(!writeSet.isEmpty()) {
-        if(writeSet.get(0).txnId == 0) {
-          //Write operation always start a txn
-          throw new IllegalStateException("Found Write lock for " + JavaUtils.lockIdToString(extLockId) + " but no txnid");
-        }
-        stmt = dbConn.createStatement();
-        StringBuilder sb = new StringBuilder(" \"WS_DATABASE\", \"WS_TABLE\", \"WS_PARTITION\", " +
-          "\"WS_TXNID\", \"WS_COMMIT_ID\" " +
-          "FROM \"WRITE_SET\" WHERE WS_COMMIT_ID >= " + writeSet.get(0).txnId + " AND (");//see commitTxn() for more info on this inequality
-        for(LockInfo info : writeSet) {
-          sb.append("(\"WS_DATABASE\" = ").append(quoteString(info.db)).append(" AND \"WS_TABLE\" = ")
-            .append(quoteString(info.table)).append(" AND \"WS_PARTITION\" ")
-            .append(info.partition == null ? "IS NULL" : "= " + quoteString(info.partition)).append(") OR ");
-        }
-        sb.setLength(sb.length() - 4);//nuke trailing " or "
-        sb.append(")");
-        //1 row is sufficient to know we have to kill the query
-        rs = stmt.executeQuery(sqlGenerator.addLimitClause(1, sb.toString()));
-        if(rs.next()) {
-          /**
-           * if here, it means we found an already committed txn which overlaps with the current one and
-           * it updated the same resource the current txn wants to update.  By First-committer-wins
-           * rule, current txn will not be allowed to commit so  may as well kill it now;  This is just an
-           * optimization to prevent wasting cluster resources to run a query which is known to be DOA.
-           * {@link #commitTxn(CommitTxnRequest)} has the primary responsibility to ensure this.
-           * checkLock() runs at READ_COMMITTED so you could have another (Hive) txn running commitTxn()
-           * in parallel and thus writing to WRITE_SET.  commitTxn() logic is properly mutexed to ensure
-           * that we don't "miss" any WW conflicts. We could've mutexed the checkLock() and commitTxn()
-           * as well but this reduces concurrency for very little gain.
-           * Note that update/delete (which runs as dynamic partition insert) acquires a lock on the table,
-           * but WRITE_SET has entries for actual partitions updated.  Thus this optimization will "miss"
-           * the WW conflict but it will be caught in commitTxn() where actual partitions written are known.
-           * This is OK since we want 2 concurrent updates that update different sets of partitions to both commit.
-           */
-          String resourceName = rs.getString(1) + '/' + rs.getString(2);
-          String partName = rs.getString(3);
-          if(partName != null) {
-            resourceName += '/' + partName;
-          }
-
-          String msg = "Aborting " + JavaUtils.txnIdToString(writeSet.get(0).txnId) +
-            " since a concurrent committed transaction [" + JavaUtils.txnIdToString(rs.getLong(4)) + "," + rs.getLong(5) +
-            "] has already updated resource '" + resourceName + "'";
-          LOG.info(msg);
-          if (abortTxns(dbConn, Collections.singletonList(writeSet.get(0).txnId), false, false,
-                  TxnErrorMsg.ABORT_CONCURRENT) != 1) {
-            throw new IllegalStateException(msg + " FAILED!");
-          }
-          dbConn.commit();
-          throw new TxnAbortedException(msg);
-        }
-        close(rs, stmt, null);
-      }
-
-      String queryStr =
-        " \"EX\".*, \"REQ\".\"HL_LOCK_INT_ID\" \"LOCK_INT_ID\", \"REQ\".\"HL_LOCK_TYPE\" \"LOCK_TYPE\" FROM (" +
-            " SELECT \"HL_LOCK_EXT_ID\", \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_STATE\", \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" < " + extLockId + ") \"EX\"" +
-        " INNER JOIN (" +
-            " SELECT \"HL_LOCK_INT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\"," +
-                " \"HL_LOCK_TYPE\" FROM \"HIVE_LOCKS\"" +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + ") \"REQ\"" +
-        " ON \"EX\".\"HL_DB\" = \"REQ\".\"HL_DB\"" +
-            " AND (\"EX\".\"HL_TABLE\" IS NULL OR \"REQ\".\"HL_TABLE\" IS NULL" +
-                " OR \"EX\".\"HL_TABLE\" = \"REQ\".\"HL_TABLE\"" +
-                " AND (\"EX\".\"HL_PARTITION\" IS NULL OR \"REQ\".\"HL_PARTITION\" IS NULL" +
-                " OR \"EX\".\"HL_PARTITION\" = \"REQ\".\"HL_PARTITION\"))" +
-        /*different locks from same txn should not conflict with each other,
-          txnId=0 means it's a select or IUD which does not write to ACID table*/
-        " WHERE (\"REQ\".\"HL_TXNID\" = 0 OR \"EX\".\"HL_TXNID\" != \"REQ\".\"HL_TXNID\")" +
-            " AND ";
-
-      /*EXCLUSIVE lock on partition should prevent SHARED_READ on the table, however there is no reason
-        for an EXCLUSIVE on a table to prevent SHARED_READ on a database. Similarly, EXCLUSIVE on a partition
-        should not conflict with SHARED_READ on a database.
-        SHARED_READ is usually acquired on a database to make sure it's not dropped, while some operation
-        is performed on that db (e.g. show tables, created table, etc).
-        EXCLUSIVE on an object may mean it's being dropped or overwritten.*/
-      String[] whereStr = {
-        // shared-read
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedRead() + " AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.exclusive() + " AND NOT (\"EX\".\"HL_TABLE\" IS NOT NULL AND \"REQ\".\"HL_TABLE\" IS NULL)",
-        // exclusive
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclusive() +
-        " AND NOT (\"EX\".\"HL_TABLE\" IS NULL AND \"EX\".\"HL_LOCK_TYPE\"=" +
-          LockTypeUtil.sharedRead() + " AND \"REQ\".\"HL_TABLE\" IS NOT NULL)",
-        // shared-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.sharedWrite() + " AND \"EX\".\"HL_LOCK_TYPE\" IN (" +
-          LockTypeUtil.exclWrite() + "," + LockTypeUtil.exclusive() + ")",
-        // excl-write
-        " \"REQ\".\"HL_LOCK_TYPE\"=" + LockTypeUtil.exclWrite() + " AND \"EX\".\"HL_LOCK_TYPE\"!=" +
-          LockTypeUtil.sharedRead()
-      };
-
-      List<String> subQuery = new ArrayList<>();
-      for (String subCond : whereStr) {
-        subQuery.add("(" + sqlGenerator.addLimitClause(1, queryStr + subCond) + ")");
-      }
-      String query = String.join(" UNION ALL ", subQuery);
-
-      stmt = dbConn.createStatement();
-      LOG.debug("Going to execute query <{}>", query);
-      rs = stmt.executeQuery(query);
-
-      if (rs.next()) {
-        // We acquire all locks for a given query atomically; if 1 blocks, all remain in Waiting state.
-        LockInfo blockedBy = new LockInfo(rs);
-        long intLockId = rs.getLong("LOCK_INT_ID");
-        char lockChar = rs.getString("LOCK_TYPE").charAt(0);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Failure to acquire lock({} intLockId:{} {}), blocked by ({})", JavaUtils.lockIdToString(extLockId),
-              intLockId, JavaUtils.txnIdToString(txnId), blockedBy);
-        }
-
-        LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-            .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-        
-        if ((zeroWaitReadEnabled && LockType.SHARED_READ == lockType || isExclusiveCTAS) && isValidTxn(txnId)) {
-          String cleanupQuery = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-          LOG.debug("Going to execute query: <{}>", cleanupQuery);
-          stmt.executeUpdate(cleanupQuery);
-          dbConn.commit();
-
-          response.setErrorMessage(String.format(
-              isExclusiveCTAS ? EXCL_CTAS_ERR_MSG : ZERO_WAIT_READ_ERR_MSG, blockedBy));
-          response.setState(LockState.NOT_ACQUIRED);
-          return response;
-        }
-        String updateBlockedByQuery = "UPDATE \"HIVE_LOCKS\"" +
-            " SET \"HL_BLOCKEDBY_EXT_ID\" = " + blockedBy.extLockId +
-            ", \"HL_BLOCKEDBY_INT_ID\" = " + blockedBy.intLockId +
-            " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND \"HL_LOCK_INT_ID\" = " + intLockId;
-
-        LOG.debug("Going to execute query: <{}>", updateBlockedByQuery);
-        int updCnt = stmt.executeUpdate(updateBlockedByQuery);
-
-        if (updCnt != 1) {
-          LOG.error("Failure to update lock (extLockId={}, intLockId={}) with the blocking lock's IDs " +
-              "(extLockId={}, intLockId={})", extLockId, intLockId, blockedBy.extLockId, blockedBy.intLockId);
-          shouldNeverHappen(txnId, extLockId, intLockId);
-        }
-        dbConn.commit();
-
-        response.setState(LockState.WAITING);
-        return response;
-      }
-      // If here, there were no locks that would block any item from 'locksBeingChecked' - acquire them all
-      acquire(dbConn, stmt, locksBeingChecked);
-
-      // We acquired all the locks, so commit and return acquired.
-      LOG.debug("Successfully acquired locks: {}", locksBeingChecked);
-      dbConn.commit();
-      response.setState(LockState.ACQUIRED);
-    } finally {
-      close(rs, stmt, null);
-    }
-    return response;
+  @RetrySemantics.ReadOnly
+  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
+      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetLatestCommittedCompactionInfoHandler(rqst));
   }
 
-  private void acquire(Connection dbConn, Statement stmt, List<LockInfo> locksBeingChecked)
-    throws SQLException, NoSuchLockException, MetaException {
-    if (locksBeingChecked == null || locksBeingChecked.isEmpty()) {
-      return;
-    }
-    long txnId = locksBeingChecked.get(0).txnId;
-    long extLockId = locksBeingChecked.get(0).extLockId;
-    String s = "UPDATE \"HIVE_LOCKS\" SET \"HL_LOCK_STATE\" = '" + LOCK_ACQUIRED + "', " +
-      //if lock is part of txn, heartbeat info is in txn record
-      "\"HL_LAST_HEARTBEAT\" = " + (isValidTxn(txnId) ? 0 : getEpochFn(dbProduct)) +
-      ",\"HL_ACQUIRED_AT\" = " + getEpochFn(dbProduct) +
-      ",\"HL_BLOCKEDBY_EXT_ID\"=NULL,\"HL_BLOCKEDBY_INT_ID\"=NULL" +
-      " WHERE \"HL_LOCK_EXT_ID\" = " +  extLockId;
-    LOG.debug("Going to execute update <{}>", s);
-    int rc = stmt.executeUpdate(s);
-    if (rc < locksBeingChecked.size()) {
-      LOG.error("Failure to acquire all locks (acquired: {}, total needed: {}).", rc, locksBeingChecked.size());
-      dbConn.rollback();
-      /*select all locks for this ext ID and see which ones are missing*/
-      String errorMsgTemplate = "No such lock(s): (%s: %s) %s";
-      Set<String> notFoundIds = locksBeingChecked.stream()
-              .map(lockInfo -> Long.toString(lockInfo.intLockId))
-              .collect(Collectors.toSet());
-      String getIntIdsQuery = "SELECT \"HL_LOCK_INT_ID\" FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute query: <{}>", getIntIdsQuery);
-      try (ResultSet rs = stmt.executeQuery(getIntIdsQuery)) {
-        while (rs.next()) {
-          notFoundIds.remove(rs.getString(1));
-        }
-      }
-      String errorMsg = String.format(errorMsgTemplate,
-              JavaUtils.lockIdToString(extLockId), String.join(", ", notFoundIds), JavaUtils.txnIdToString(txnId));
-      throw new NoSuchLockException(errorMsg);
-    }
+  public MetricsInfo getMetricsInfo() throws MetaException {
+    int threshold = MetastoreConf.getIntVar(conf, MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD);
+    MetricsInfo metrics = jdbcResource.execute(MetricsInfoHandler.INSTANCE);
+    Set<String> resourceNames = jdbcResource.execute(new TablesWithAbortedTxnsHandler(threshold));
+    metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
+    metrics.setTablesWithXAbortedTxns(resourceNames);    
+    return metrics;
   }
 
-  /**
-   * Heartbeats on the lock table.  This commits, so do not enter it with any state.
-   * Should not be called on a lock that belongs to transaction.
-   */
-  private void heartbeatLock(Connection dbConn, long extLockId)
-    throws NoSuchLockException, SQLException, MetaException {
-    // If the lock id is 0, then there are no locks in this heartbeat
-    if (extLockId == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String updateHeartbeatQuery = "UPDATE \"HIVE_LOCKS\" SET \"HL_LAST_HEARTBEAT\" = " +
-          getEpochFn(dbProduct) + " WHERE \"HL_LOCK_EXT_ID\" = " + extLockId;
-      LOG.debug("Going to execute update <{}>", updateHeartbeatQuery);
-      int rc = stmt.executeUpdate(updateHeartbeatQuery);
-      if (rc < 1) {
-        LOG.error("Failure to update last heartbeat for extLockId={}.", extLockId);
-        dbConn.rollback();
-        throw new NoSuchLockException("No such lock: " + JavaUtils.lockIdToString(extLockId));
-      }
-      LOG.debug("Successfully heartbeated for extLockId={}", extLockId);
-      dbConn.commit();
-    }
-  }
 
-  // Heartbeats on the txn table.  This commits, so do not enter it with any state
-  private void heartbeatTxn(Connection dbConn, long txnid)
-    throws NoSuchTxnException, TxnAbortedException, SQLException, MetaException {
-    // If the txnid is 0, then there are no transactions in this heartbeat
-    if (txnid == 0) {
-      return;
-    }
-    try (Statement stmt = dbConn.createStatement()) {
-      String s = "UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-          " WHERE \"TXN_ID\" = " + txnid + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-      LOG.debug("Going to execute update <{}>", s);
-      int rc = stmt.executeUpdate(s);
-      if (rc < 1) {
-        ensureValidTxn(dbConn, txnid, stmt); // This should now throw some useful exception.
-        LOG.error("Can neither heartbeat txn (txnId={}) nor confirm it as invalid.", txnid);
-        dbConn.rollback();
-        throw new NoSuchTxnException("No such txn: " + txnid);
-      }
-      LOG.debug("Successfully heartbeated for txnId={}", txnid);
-      dbConn.commit();
-    }
+  private static void shouldNeverHappen(long txnid) {
+    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
   }
 
   /**
-   * Returns the state of the transaction if it's able to determine it. Some cases where it cannot:
-   * 1. txnid was Aborted/Committed and then GC'd (compacted)
-   * 2. txnid was committed but it didn't modify anything (nothing in COMPLETED_TXN_COMPONENTS)
+   * Retry-by-caller note:
+   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
+   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
    */
-  private TxnStatus findTxnState(long txnid, Statement stmt) throws SQLException, MetaException {
-    String s = "SELECT \"TXN_STATE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnid;
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        s =
-            sqlGenerator.addLimitClause(1, "1 FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_TXNID\" = "
-                + txnid);
-        LOG.debug("Going to execute query <{}>", s);
-        try (ResultSet rs2 = stmt.executeQuery(s)) {
-          if (rs2.next()) {
-            return TxnStatus.COMMITTED;
-          }
-        }
-        // could also check WRITE_SET but that seems overkill
-        return TxnStatus.UNKNOWN;
-      }
-      return TxnStatus.fromString(rs.getString(1));
-    }
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void addDynamicPartitions(AddDynamicPartitions rqst) throws NoSuchTxnException,  TxnAbortedException, MetaException {
+    TxnType txnType = jdbcResource.execute(new GetOpenTxnTypeAndLockHandler(sqlGenerator, rqst.getTxnid()));
+    if (txnType == null) {
+      //ensures txn is still there and in expected state
+      new EnsureValidTxnFunction(rqst.getTxnid()).execute(jdbcResource);
+      shouldNeverHappen(rqst.getTxnid());
+    }
+    jdbcResource.execute(new InsertTxnComponentsCommand(rqst));
+    jdbcResource.getJdbcTemplate().update("DELETE FROM \"TXN_COMPONENTS\" " +
+            "WHERE \"TC_TXNID\" = :txnId AND \"TC_DATABASE\" = :dbName AND \"TC_TABLE\" = :tableName AND \"TC_PARTITION\" IS NULL",
+        new MapSqlParameterSource()
+            .addValue("txnId", rqst.getTxnid())
+            .addValue("dbName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getDbname()))
+            .addValue("tableName", org.apache.commons.lang3.StringUtils.lowerCase(rqst.getTablename())));
   }
 
   /**
-   * Checks if all the txns in the list are in open state and not read-only.
-   * @param txnIds list of txns to be evaluated for open state/read-only status
-   * @param stmt db statement
-   * @return If all the txns in open state and not read-only, then return true else false
+   * Clean up corresponding records in metastore tables when corresponding object is dropped,
+   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
+   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
+   * operations.
+   * <p>
+   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
    */
-  private boolean isTxnsOpenAndNotReadOnly(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Get the count of txns from the given list that are in open state and not read-only.
-    // If the returned count is same as the input number of txns, then all txns are in open state and not read-only.
-    prefix.append("SELECT COUNT(*) FROM \"TXNS\" WHERE \"TXN_STATE\" = " + TxnStatus.OPEN
-        + " AND \"TXN_TYPE\" != " + TxnType.READ_ONLY.getValue() + " AND ");
-
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-
-    long count = 0;
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        if (rs.next()) {
-          count += rs.getLong(1);
-        }
-      }
-    }
-    return count == txnIds.size();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), keepTxnToWriteIdMetaData, null)
+        .execute(jdbcResource);
   }
 
-  /**
-   * Get txns from the list that are either aborted or read-only.
-   * @param txnIds list of txns to be evaluated for aborted state/read-only status
-   * @param stmt db statement
-   */
-  private String getAbortedAndReadOnlyTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are either aborted or read-only.
-    prefix.append("SELECT \"TXN_ID\", \"TXN_STATE\", \"TXN_TYPE\" FROM \"TXNS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"TXN_ID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          TxnStatus txnState = TxnStatus.fromString(rs.getString(2));
-          TxnType txnType = TxnType.findByValue(rs.getInt(3));
-
-          if (txnState != TxnStatus.OPEN) {
-            txnInfo.append("{").append(txnId).append(",").append(txnState).append("}");
-          } else if (txnType == TxnType.READ_ONLY) {
-            txnInfo.append("{").append(txnId).append(",read-only}");
-          }
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  @RetrySemantics.Idempotent
+  public void cleanupRecords(HiveObjectType type, Database db, Table table,
+        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
+    new CleanupRecordsFunction(type, db, table, partitionIterator, getDefaultCatalog(conf), false, txnId)
+        .execute(jdbcResource);
   }
-
+  
   /**
-   * Get txns from the list that are committed.
-   * @param txnIds list of txns to be evaluated for committed state
-   * @param stmt db statement
+   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
    */
-  private String getCommittedTxns(List<Long> txnIds, Statement stmt) throws SQLException {
-    List<String> queries = new ArrayList<>();
-    StringBuilder prefix = new StringBuilder();
-
-    // Check if any of the txns in the list are committed.
-    prefix.append("SELECT \"CTC_TXNID\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE ");
-    TxnUtils.buildQueryWithINClause(conf, queries, prefix, new StringBuilder(),
-        txnIds, "\"CTC_TXNID\"", false, false);
-    StringBuilder txnInfo = new StringBuilder();
-
-    for (String query : queries) {
-      LOG.debug("Going to execute query <{}>", query);
-      try (ResultSet rs = stmt.executeQuery(query)) {
-        while (rs.next()) {
-          long txnId = rs.getLong(1);
-          txnInfo.append("{").append(txnId).append(",c}");
-        }
-      }
-    }
-    return txnInfo.toString();
+  @Override
+  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
+      String newCatName, String newDbName, String newTabName, String newPartName)
+      throws MetaException {
+    new OnRenameFunction(oldCatName, oldDbName, oldTabName, oldPartName, newCatName, newDbName, newTabName, newPartName).execute(jdbcResource);
   }
-
   /**
-   * Used to raise an informative error when the caller expected a txn in a particular TxnStatus
-   * but found it in some other status
+   * For testing only, do not use.
    */
-  private static void raiseTxnUnexpectedState(TxnStatus actualStatus, long txnid)
-    throws NoSuchTxnException, TxnAbortedException {
-    switch (actualStatus) {
-      case ABORTED:
-        throw new TxnAbortedException("Transaction " + JavaUtils.txnIdToString(txnid) + " already aborted");
-      case COMMITTED:
-        throw new NoSuchTxnException("Transaction " + JavaUtils.txnIdToString(txnid) + " is already committed.");
-      case UNKNOWN:
-        throw new NoSuchTxnException("No such transaction " + JavaUtils.txnIdToString(txnid));
-      case OPEN:
-        throw new NoSuchTxnException(JavaUtils.txnIdToString(txnid) + " is " + TxnStatus.OPEN);
-      default:
-        throw new IllegalArgumentException("Unknown TxnStatus " + actualStatus);
-    }
+  @VisibleForTesting
+  public int numLocksInLockTable() {
+    return Objects.requireNonNull(
+        jdbcResource.getJdbcTemplate().queryForObject("SELECT COUNT(*) FROM \"HIVE_LOCKS\"", new MapSqlParameterSource(), Integer.TYPE),
+        "This never should be null, it's just to suppress warnings");
   }
+
   /**
-   * Returns the state of the transaction with {@code txnid} or throws if {@code raiseError} is true.
+   * For testing only, do not use.

Review Comment:
   ok, could we please add TODO, so we'll this all todo stuff in a report later



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1425276096


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -648,260 +400,85 @@ public OpenTxnsResponse openTxns(OpenTxnRequest rqst) throws MetaException {
       throw new MetaException("Invalid input for number of txns: " + numTxns);
     }
 
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        /*
-         * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
-         * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
-         * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
-         * performance degradation with high transaction load.
-         * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
-         * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
-         * the period.
-         * Example: At t[0] there is one open transaction in the TXNS table, T[1].
-         * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
-         * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
-         * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
-         * open transaction list. T[1] because it is presented as open in TXNS,
-         * T[2] because it is a missing sequence.
-         *
-         * In the current design, there can be several metastore instances running in a given Warehouse.
-         * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
-         * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
-         * Now the same client will start another transaction, except it ends up on MS2 and may get
-         * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
-         * on read will thing the version of the row from transaction ID 500 is the latest one.
-         *
-         * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
-         * set could support a write-through cache for added performance.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
-         * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
-         * Since table locks are working in Derby, we don't need the lockInternal call here.
-         * Example: Suppose we have two transactions with update like x = x+1.
-         * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
-         * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
-         * and we will have a lost update problem
-         */
-        acquireTxnLock(stmt, true);
-        // Measure the time from acquiring the sequence value, till committing in the TXNS table
-        StopWatch generateTransactionWatch = new StopWatch();
-        generateTransactionWatch.start();
-
-        List<Long> txnIds = openTxns(dbConn, rqst);
+    /*
+     * To make {@link #getOpenTxns()}/{@link #getOpenTxnsInfo()} work correctly, this operation must ensure
+     * that looking at the TXNS table every open transaction could be identified below a given High Water Mark.
+     * One way to do it, would be to serialize the openTxns call with a S4U lock, but that would cause
+     * performance degradation with high transaction load.
+     * To enable parallel openTxn calls, we define a time period (TXN_OPENTXN_TIMEOUT) and consider every
+     * transaction missing from the TXNS table in that period open, and prevent opening transaction outside
+     * the period.
+     * Example: At t[0] there is one open transaction in the TXNS table, T[1].
+     * T[2] acquires the next sequence at t[1] but only commits into the TXNS table at t[10].
+     * T[3] acquires its sequence at t[2], and commits into the TXNS table at t[3].
+     * Then T[3] calculates it’s snapshot at t[4] and puts T[1] and also T[2] in the snapshot’s
+     * open transaction list. T[1] because it is presented as open in TXNS,
+     * T[2] because it is a missing sequence.
+     *
+     * In the current design, there can be several metastore instances running in a given Warehouse.
+     * This makes ideas like reserving a range of IDs to save trips to DB impossible.  For example,
+     * a client may go to MS1 and start a transaction with ID 500 to update a particular row.
+     * Now the same client will start another transaction, except it ends up on MS2 and may get
+     * transaction ID 400 and update the same row.  Now the merge that happens to materialize the snapshot
+     * on read will thing the version of the row from transaction ID 500 is the latest one.
+     *
+     * Longer term we can consider running Active-Passive MS (at least wrt to ACID operations).  This
+     * set could support a write-through cache for added performance.
+     */
+    /*
+     * The openTxn and commitTxn must be mutexed, when committing a not read only transaction.
+     * This is achieved by requesting a shared table lock here, and an exclusive one at commit.
+     * Since table locks are working in Derby, we don't need the lockInternal call here.
+     * Example: Suppose we have two transactions with update like x = x+1.
+     * We have T[3,3] that was using a value from a snapshot with T[2,2]. If we allow committing T[3,3]
+     * and opening T[4] parallel it is possible, that T[4] will be using the value from a snapshot with T[2,2],
+     * and we will have a lost update problem
+     */
+    acquireTxnLock(true);
+    // Measure the time from acquiring the sequence value, till committing in the TXNS table
+    StopWatch generateTransactionWatch = new StopWatch();
+    generateTransactionWatch.start();
 
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        generateTransactionWatch.stop();
-        long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
-        TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-        if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
-          /*
-           * The commit was too slow, we can not allow this to continue (except if it is read only,
-           * since that can not cause dirty reads).
-           * When calculating the snapshot for a given transaction, we look back for possible open transactions
-           * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
-           * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
-           * because there can be other transactions running, that didn't considered this transactionId open,
-           * this could cause dirty reads.
-           */
-          LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
-          deleteInvalidOpenTransactions(dbConn, txnIds);
-          dbConn.commit();
-          /*
-           * We do not throw RetryException directly, to not circumvent the max retry limit
-           */
-          throw new SQLException("OpenTxnTimeOut exceeded", MANUAL_RETRY);
-        }
-        return new OpenTxnsResponse(txnIds);
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "openTxns(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return openTxns(rqst);
-    }
-  }
+    List<Long> txnIds = new OpenTxnsFunction(rqst, openTxnTimeOutMillis, transactionalListeners).execute(jdbcResource);
 
-  private List<Long> openTxns(Connection dbConn, OpenTxnRequest rqst)
-          throws SQLException, MetaException {
-    int numTxns = rqst.getNum_txns();
-    // Make sure the user has not requested an insane amount of txns.
-    int maxTxns = MetastoreConf.getIntVar(conf, ConfVars.TXN_MAX_OPEN_BATCH);
-    if (numTxns > maxTxns) {
-      numTxns = maxTxns;
-    }
-    List<PreparedStatement> insertPreparedStmts = null;
+    LOG.debug("Going to commit");
+    jdbcResource.getTransactionManager().getActiveTransaction().createSavepoint();
+    generateTransactionWatch.stop();
+    long elapsedMillis = generateTransactionWatch.getTime(TimeUnit.MILLISECONDS);
     TxnType txnType = rqst.isSetTxn_type() ? rqst.getTxn_type() : TxnType.DEFAULT;
-    boolean isReplayedReplTxn = txnType == TxnType.REPL_CREATED;
-    boolean isHiveReplTxn = rqst.isSetReplPolicy() && txnType == TxnType.DEFAULT;
-    try {
-      if (isReplayedReplTxn) {
-        assert rqst.isSetReplPolicy();
-        List<Long> targetTxnIdList = getTargetTxnIdList(rqst.getReplPolicy(), rqst.getReplSrcTxnIds(), dbConn);
-
-        if (!targetTxnIdList.isEmpty()) {
-          if (targetTxnIdList.size() != rqst.getReplSrcTxnIds().size()) {
-            LOG.warn("target txn id number {} is not matching with source txn id number {}",
-                targetTxnIdList, rqst.getReplSrcTxnIds());
-          }
-          LOG.info("Target transactions {} are present for repl policy : {} and Source transaction id : {}",
-              targetTxnIdList.toString(), rqst.getReplPolicy(), rqst.getReplSrcTxnIds().toString());
-          return targetTxnIdList;
-        }
-      }
-
-      long minOpenTxnId = 0;
-      if (useMinHistoryLevel) {
-        minOpenTxnId = getMinOpenTxnIdWaterMark(dbConn);
-      }
-
-      List<Long> txnIds = new ArrayList<>(numTxns);
+    if (txnType != TxnType.READ_ONLY && elapsedMillis >= openTxnTimeOutMillis) {
       /*
-       * The getGeneratedKeys are not supported in every dbms, after executing a multi line insert.
-       * But it is supported in every used dbms for single line insert, even if the metadata says otherwise.
-       * If the getGeneratedKeys are not supported first we insert a random batchId in the TXN_META_INFO field,
-       * then the keys are selected beck with that batchid.
+       * The commit was too slow, we can not allow this to continue (except if it is read only,
+       * since that can not cause dirty reads).
+       * When calculating the snapshot for a given transaction, we look back for possible open transactions
+       * (that are not yet committed in the TXNS table), for TXN_OPENTXN_TIMEOUT period.
+       * We can not allow a write transaction, that was slower than TXN_OPENTXN_TIMEOUT to continue,
+       * because there can be other transactions running, that didn't considered this transactionId open,
+       * this could cause dirty reads.
        */
-      boolean genKeySupport = dbProduct.supportsGetGeneratedKeys();
-      genKeySupport = genKeySupport || (numTxns == 1);
-
-      String insertQuery = String.format(TXNS_INSERT_QRY, getEpochFn(dbProduct), getEpochFn(dbProduct));
-      LOG.debug("Going to execute insert <{}>", insertQuery);
-      try (PreparedStatement ps = dbConn.prepareStatement(insertQuery, new String[] {"TXN_ID"})) {
-        String state = genKeySupport ? TxnStatus.OPEN.getSqlConst() : TXN_TMP_STATE;
-        if (numTxns == 1) {
-          ps.setString(1, state);
-          ps.setString(2, rqst.getUser());
-          ps.setString(3, rqst.getHostname());
-          ps.setInt(4, txnType.getValue());
-          txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, false));
-        } else {
-          for (int i = 0; i < numTxns; ++i) {
-            ps.setString(1, state);
-            ps.setString(2, rqst.getUser());
-            ps.setString(3, rqst.getHostname());
-            ps.setInt(4, txnType.getValue());
-            ps.addBatch();
-
-            if ((i + 1) % maxBatchSize == 0) {
-              txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, true));
-            }
-          }
-          if (numTxns % maxBatchSize != 0) {
-            txnIds.addAll(executeTxnInsertBatchAndExtractGeneratedKeys(dbConn, genKeySupport, ps, true));
-          }
-        }
-      }
-
-      assert txnIds.size() == numTxns;
-
-      addTxnToMinHistoryLevel(dbConn, txnIds, minOpenTxnId);
-
-      if (isReplayedReplTxn) {
-        List<String> rowsRepl = new ArrayList<>(numTxns);
-        List<String> params = Collections.singletonList(rqst.getReplPolicy());
-        List<List<String>> paramsList = new ArrayList<>(numTxns);
-        for (int i = 0; i < numTxns; i++) {
-          rowsRepl.add("?," + rqst.getReplSrcTxnIds().get(i) + "," + txnIds.get(i));
-          paramsList.add(params);
-        }
-
-        insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                "\"REPL_TXN_MAP\" (\"RTM_REPL_POLICY\", \"RTM_SRC_TXN_ID\", \"RTM_TARGET_TXN_ID\")", rowsRepl,
-                paramsList);
-        for (PreparedStatement pst : insertPreparedStmts) {
-          pst.execute();
-        }
-      }
-
-      if (transactionalListeners != null && !isHiveReplTxn) {
-        MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                EventMessage.EventType.OPEN_TXN, new OpenTxnEvent(txnIds, txnType), dbConn, sqlGenerator);
-      }
-      return txnIds;
-    } finally {
-      if (insertPreparedStmts != null) {
-        for (PreparedStatement pst : insertPreparedStmts) {
-          pst.close();
-        }
-      }
-    }
-  }
+      LOG.error("OpenTxnTimeOut exceeded commit duration {}, deleting transactionIds: {}", elapsedMillis, txnIds);
 
-  private List<Long> executeTxnInsertBatchAndExtractGeneratedKeys(Connection dbConn, boolean genKeySupport,
-      PreparedStatement ps, boolean batch) throws SQLException {
-    List<Long> txnIds = new ArrayList<>();
-    if (batch) {
-      ps.executeBatch();
-    } else {
-      // For slight performance advantage we do not use the executeBatch, when we only have one row
-      ps.execute();
-    }
-    if (genKeySupport) {
-      try (ResultSet generatedKeys = ps.getGeneratedKeys()) {
-        while (generatedKeys.next()) {
-          txnIds.add(generatedKeys.getLong(1));
-        }
-      }
-    } else {
-      try (PreparedStatement pstmt =
-          dbConn.prepareStatement("SELECT \"TXN_ID\" FROM \"TXNS\" WHERE \"TXN_STATE\" = ?")) {
-        pstmt.setString(1, TXN_TMP_STATE);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          while (rs.next()) {
-            txnIds.add(rs.getLong(1));
-          }
+      if (!txnIds.isEmpty()) {
+        try {
+          sqlRetryHandler.executeWithRetry(new SqlRetryCallProperties().withCallerId("deleteInvalidOpenTransactions"),
+              () -> {
+                jdbcResource.execute(new DeleteInvalidOpenTxnsCommand(txnIds));
+                LOG.info("Removed transactions: ({}) from TXNS", txnIds);
+                jdbcResource.execute(new RemoveTxnsFromMinHistoryLevelCommand(txnIds));
+                return null;
+              });

Review Comment:
   Yes, openTxns() has the `Transactional` annotation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418845605


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TransactionalRetryProxy.java:
##########
@@ -149,15 +164,23 @@ public Object invoke(Object proxy, Method method, Object[] args) throws Throwabl
           LOG.debug("Successfull method invocation within retry context: {}", callerId);
           return result;
         } catch (IllegalAccessException | InvocationTargetException | UndeclaredThrowableException e) {
-          if (e.getCause() instanceof MetaException) {
-            throw (MetaException) e.getCause();
+          if (e.getCause() instanceof TException) {
+            throw (TException) e.getCause();
           } else if (e.getCause() instanceof RuntimeException) {
             throw (RuntimeException) e.getCause();
           } else {
             throw new RuntimeException(e);
           }
-        } catch (Throwable e) {
-          throw new RuntimeException(e);
+        } catch (TException | DataAccessException e) {

Review Comment:
   should we extract to rethrowException(e) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418687026


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -193,10 +194,11 @@ public boolean isDeadlock(SQLException e) {
 
   /**
    * Is the given exception a table not found exception
-   * @param e Exception
+   * @param ex Exception
    * @return
    */
-  public boolean isTableNotExistsError(SQLException e) {
+  public boolean isTableNotExistsError(Throwable ex) {

Review Comment:
   why is this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418913120


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/functions/HeartBeatLockFunction.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.functions;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionalFunction;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnUtils.getEpochFn;
+
+/**
+ * Heartbeats on the lock table.  This commits, so do not enter it with any state.
+ * Should not be called on a lock that belongs to transaction.
+ */
+public class HeartBeatLockFunction implements TransactionalFunction<Void> {

Review Comment:
   HeartbeatLockFunction



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419085985


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1740,4039 +649,413 @@ public long getLatestTxnIdInConflict(long txnid) throws MetaException {
    * @param txnId
    * @throws MetaException
    */
-    private List<String> getTxnDbsUpdated(long txnId, Connection dbConn) throws MetaException {
+  private List<String> getTxnDbsUpdated(long txnId) throws MetaException {
     try {
-      try (Statement stmt = dbConn.createStatement()) {
-
-        String query = "SELECT DISTINCT \"T2W_DATABASE\" " +
-                " FROM \"TXN_TO_WRITE_ID\" \"COMMITTED\"" +
-                "   WHERE \"T2W_TXNID\" = " + txnId;
-
-        LOG.debug("Going to execute query: <{}>", query);
-        try (ResultSet rs = stmt.executeQuery(query)) {
-          List<String> dbsUpdated = new ArrayList<String>();
-          while (rs.next()) {
-            dbsUpdated.add(rs.getString(1));
-          }
-          return dbsUpdated;
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnDbsUpdated");
-        throw new MetaException(StringUtils.stringifyException(e));
-      }
-    } catch (RetryException e) {
-      return getTxnDbsUpdated(txnId, dbConn);
-    }
-  }
-
-
-  private ResultSet checkForWriteConflict(Statement stmt, long txnid) throws SQLException, MetaException {
-    String writeConflictQuery = sqlGenerator.addLimitClause(1, "\"COMMITTED\".\"WS_TXNID\", \"COMMITTED\".\"WS_COMMIT_ID\", " +
-            "\"COMMITTED\".\"WS_DATABASE\", \"COMMITTED\".\"WS_TABLE\", \"COMMITTED\".\"WS_PARTITION\", " +
-            "\"CUR\".\"WS_COMMIT_ID\" \"CUR_WS_COMMIT_ID\", \"CUR\".\"WS_OPERATION_TYPE\" \"CUR_OP\", " +
-            "\"COMMITTED\".\"WS_OPERATION_TYPE\" \"COMMITTED_OP\" FROM \"WRITE_SET\" \"COMMITTED\" INNER JOIN \"WRITE_SET\" \"CUR\" " +
-            "ON \"COMMITTED\".\"WS_DATABASE\"=\"CUR\".\"WS_DATABASE\" AND \"COMMITTED\".\"WS_TABLE\"=\"CUR\".\"WS_TABLE\" " +
-            //For partitioned table we always track writes at partition level (never at table)
-            //and for non partitioned - always at table level, thus the same table should never
-            //have entries with partition key and w/o
-            "AND (\"COMMITTED\".\"WS_PARTITION\"=\"CUR\".\"WS_PARTITION\" OR (\"COMMITTED\".\"WS_PARTITION\" IS NULL AND \"CUR\".\"WS_PARTITION\" IS NULL)) " +
-            "WHERE \"CUR\".\"WS_TXNID\" <= \"COMMITTED\".\"WS_COMMIT_ID\"" + //txns overlap; could replace ws_txnid
-            // with txnid, though any decent DB should infer this
-            " AND \"CUR\".\"WS_TXNID\"=" + txnid + //make sure RHS of join only has rows we just inserted as
-            // part of this commitTxn() op
-            " AND \"COMMITTED\".\"WS_TXNID\" <> " + txnid + //and LHS only has committed txns
-            //U+U and U+D and D+D is a conflict and we don't currently track I in WRITE_SET at all
-            //it may seem like D+D should not be in conflict but consider 2 multi-stmt txns
-            //where each does "delete X + insert X, where X is a row with the same PK.  This is
-            //equivalent to an update of X but won't be in conflict unless D+D is in conflict.
-            //The same happens when Hive splits U=I+D early so it looks like 2 branches of a
-            //multi-insert stmt (an Insert and a Delete branch).  It also 'feels'
-            // un-serializable to allow concurrent deletes
-            " and (\"COMMITTED\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE +
-            ", " + OperationType.DELETE +
-            ") AND \"CUR\".\"WS_OPERATION_TYPE\" IN(" + OperationType.UPDATE+ ", "
-            + OperationType.DELETE + "))");
-    LOG.debug("Going to execute query: <{}>", writeConflictQuery);
-    return stmt.executeQuery(writeConflictQuery);
-  }
-
-  private void moveTxnComponentsToCompleted(Statement stmt, long txnid, char isUpdateDelete) throws SQLException {
-    // Move the record from txn_components into completed_txn_components so that the compactor
-    // knows where to look to compact.
-    String s = "INSERT INTO \"COMPLETED_TXN_COMPONENTS\" (\"CTC_TXNID\", \"CTC_DATABASE\", " +
-            "\"CTC_TABLE\", \"CTC_PARTITION\", \"CTC_WRITEID\", \"CTC_UPDATE_DELETE\") SELECT \"TC_TXNID\"," +
-        " \"TC_DATABASE\", \"TC_TABLE\", \"TC_PARTITION\", \"TC_WRITEID\", '" + isUpdateDelete +
-        "' FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid +
-        //we only track compactor activity in TXN_COMPONENTS to handle the case where the
-        //compactor txn aborts - so don't bother copying it to COMPLETED_TXN_COMPONENTS
-        " AND \"TC_OPERATION_TYPE\" <> " + OperationType.COMPACT;
-    LOG.debug("Going to execute insert <{}>", s);
-
-    if ((stmt.executeUpdate(s)) < 1) {
-      //this can be reasonable for an empty txn START/COMMIT or read-only txn
-      //also an IUD with DP that didn't match any rows.
-      LOG.info("Expected to move at least one record from txn_components to "
-          + "completed_txn_components when committing txn! {}", JavaUtils.txnIdToString(txnid));
-    }
-  }
-
-  /**
-   * See overridden method in CompactionTxnHandler also.
-   */
-  protected void updateWSCommitIdAndCleanUpMetadata(Statement stmt, long txnid, TxnType txnType,
-      Long commitId, long tempId) throws SQLException, MetaException {
-    List<String> queryBatch = new ArrayList<>(5);
-    // update write_set with real commitId
-    if (commitId != null) {
-      queryBatch.add("UPDATE \"WRITE_SET\" SET \"WS_COMMIT_ID\" = " + commitId +
-              " WHERE \"WS_COMMIT_ID\" = " + tempId + " AND \"WS_TXNID\" = " + txnid);
-    }
-    // clean up txn related metadata
-    if (txnType != TxnType.READ_ONLY) {
-      queryBatch.add("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_TXNID\" = " + txnid);
-    }
-    queryBatch.add("DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_TXNID\" = " + txnid);
-    // DO NOT remove the transaction from the TXN table, the cleaner will remove it when appropriate
-    queryBatch.add("UPDATE \"TXNS\" SET \"TXN_STATE\" = " + TxnStatus.COMMITTED + " WHERE \"TXN_ID\" = " + txnid);
-    if (txnType == TxnType.MATER_VIEW_REBUILD) {
-      queryBatch.add("DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE \"MRL_TXN_ID\" = " + txnid);
-    }
-    // execute all in one batch
-    executeQueriesInBatchNoCount(dbProduct, stmt, queryBatch, maxBatchSize);
-  }
-
-  private void updateKeyValueAssociatedWithTxn(CommitTxnRequest rqst, Statement stmt) throws SQLException {
-    if (!rqst.getKeyValue().getKey().startsWith(TxnStore.TXN_KEY_START)) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " key should start with " + TXN_KEY_START + ".";
-      LOG.warn(errorMsg);
-      throw new IllegalArgumentException(errorMsg);
-    }
-    String s = "UPDATE \"TABLE_PARAMS\" SET"
-        + " \"PARAM_VALUE\" = " + quoteString(rqst.getKeyValue().getValue())
-        + " WHERE \"TBL_ID\" = " + rqst.getKeyValue().getTableId()
-        + " AND \"PARAM_KEY\" = " + quoteString(rqst.getKeyValue().getKey());
-    LOG.debug("Going to execute update <{}>", s);
-    int affectedRows = stmt.executeUpdate(s);
-    if (affectedRows != 1) {
-      String errorMsg = "Error updating key/value in the sql backend with"
-          + " txnId=" + rqst.getTxnid() + ","
-          + " tableId=" + rqst.getKeyValue().getTableId() + ","
-          + " key=" + rqst.getKeyValue().getKey() + ","
-          + " value=" + rqst.getKeyValue().getValue() + "."
-          + " Only one row should have been affected but "
-          + affectedRows + " rows where affected.";
-      LOG.warn(errorMsg);
-      throw new IllegalStateException(errorMsg);
+      return sqlRetryHandler.executeWithRetry(
+          new SqlRetryCallProperties().withCallerId("GetTxnDbsUpdatedHandler"),
+          () -> jdbcResource.execute(new GetTxnDbsUpdatedHandler(txnId)));
+    } catch (MetaException e) {
+      throw e;
+    } catch (TException e) {
+      throw new MetaException(e.getMessage());
     }
   }
 
   /**
-   * Replicate Table Write Ids state to mark aborted write ids and writeid high water mark.
+   * Replicate Table Write Ids state to mark aborted write ids and writeid high watermark.
    * @param rqst info on table/partitions and writeid snapshot to replicate.
    * @throws MetaException
    */
   @Override
   @RetrySemantics.Idempotent("No-op if already replicated the writeid state")
   public void replTableWriteIdState(ReplTblWriteIdStateRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    ValidWriteIdList validWriteIdList = new ValidReaderWriteIdList(rqst.getValidWriteIdlist());
-
-    // Get the abortedWriteIds which are already sorted in ascending order.
-    List<Long> abortedWriteIds = getAbortedWriteIds(validWriteIdList);
-    int numAbortedWrites = abortedWriteIds.size();
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      PreparedStatement pStmt = null;
-      List<PreparedStatement> insertPreparedStmts = null;
-      ResultSet rs = null;
-      List<String> params = Arrays.asList(dbName, tblName);
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        // Check if this txn state is already replicated for this given table. If yes, then it is
-        // idempotent case and just return.
-        String sql = "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (rs.next()) {
-          LOG.info("Idempotent flow: WriteId state <{}> is already applied for the table: {}.{}", validWriteIdList,
-              dbName, tblName);
-          rollbackDBConn(dbConn);
-          return;
-        }
-
-        if (numAbortedWrites > 0) {
-          // Allocate/Map one txn per aborted writeId and abort the txn to mark writeid as aborted.
-          // We don't use the txnLock, all of these transactions will be aborted in this one rdbm transaction
-          // So they will not effect the commitTxn in any way
-          List<Long> txnIds = openTxns(dbConn,
-                  new OpenTxnRequest(numAbortedWrites, rqst.getUser(), rqst.getHostName()));
-          assert(numAbortedWrites == txnIds.size());
-
-          // Map each aborted write id with each allocated txn.
-          List<String> rows = new ArrayList<>();
-          List<List<String>> paramsList = new ArrayList<>();
-          int i = 0;
-          for (long txn : txnIds) {
-            long writeId = abortedWriteIds.get(i++);
-            rows.add(txn + ", ?, ?, " + writeId);
-            paramsList.add(params);
-            LOG.info("Allocated writeID: {} for txnId: {}", writeId, txn);
-          }
+    new ReplTableWriteIdStateFunction(rqst, mutexAPI, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Insert entries to TXN_TO_WRITE_ID for aborted write ids
-          insertPreparedStmts = sqlGenerator.createInsertValuesPreparedStmt(dbConn,
-                  "\"TXN_TO_WRITE_ID\" (\"T2W_TXNID\", \"T2W_DATABASE\", \"T2W_TABLE\", \"T2W_WRITEID\")", rows,
-                  paramsList);
-          for (PreparedStatement pst : insertPreparedStmts) {
-            pst.execute();
-          }
+  @Override
+  @RetrySemantics.ReadOnly
+  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
+    return new GetValidWriteIdsFunction(rqst, openTxnTimeOutMillis).execute(jdbcResource);
+  }
+  
+  @Override
+  @RetrySemantics.Idempotent
+  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst) throws MetaException {
+    return new AllocateTableWriteIdsFunction(rqst, transactionalListeners).execute(jdbcResource);
+  }
 
-          // Abort all the allocated txns so that the mapped write ids are referred as aborted ones.
-          int numAborts = abortTxns(dbConn, txnIds, false, false, TxnErrorMsg.ABORT_REPL_WRITEID_TXN);
-          assert(numAborts == numAbortedWrites);
-        }
+  @Override
+  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
+    return jdbcResource.execute(new GetMaxAllocatedTableWriteIdHandler(rqst));
+  }
 
-        // There are some txns in the list which has no write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // It is expected NEXT_WRITE_ID doesn't have entry for this table and hence directly insert it.
-        long nextWriteId = validWriteIdList.getHighWatermark() + 1;
+  @Override
+  public void seedWriteId(SeedTableWriteIdsRequest rqst) throws MetaException {
+    //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
+    //for this table.  It also has a unique index in case 'should not' is violated
 
-        // First allocation of write id (hwm+1) should add the table to the next_write_id meta table.
-        sql = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(nextWriteId) + ")";
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, sql, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + sql.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        pStmt.execute();
+    // First allocation of write id should add the table to the next_write_id meta table
+    // The initial value for write id should be 1 and hence we add 1 with number of write ids
+    // allocated here
+    jdbcResource.getJdbcTemplate().update(
+        "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (:db, :table, :writeId)",
+        new MapSqlParameterSource()
+            .addValue("db", rqst.getDbName())
+            .addValue("table", rqst.getTableName())
+            .addValue("writeId", rqst.getSeedWriteId() + 1));
+  }
 
-        LOG.info("WriteId state <{}> is applied for the table: {}.{}", validWriteIdList, dbName, tblName);
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "replTableWriteIdState(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        if (insertPreparedStmts != null) {
-          for (PreparedStatement pst : insertPreparedStmts) {
-            closeStmt(pst);
-          }
-        }
-        closeStmt(pStmt);
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      replTableWriteIdState(rqst);
+  @Override
+  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
+    /*
+     * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
+     * if there are concurrent open transactions
+     */
+    acquireTxnLock(false);
+    long highWaterMark = jdbcResource.execute(new GetHighWaterMarkHandler());
+    if (highWaterMark >= rqst.getSeedTxnId()) {
+      throw new MetaException(MessageFormat
+          .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
     }
+    jdbcResource.getJdbcTemplate().getJdbcTemplate().execute((Statement stmt) -> stmt.execute(dbProduct.getTxnSeedFn(rqst.getSeedTxnId())));
+  }
 
-    // Schedule Major compaction on all the partitions/table to clean aborted data
-    if (numAbortedWrites > 0) {
-      CompactionRequest compactRqst = new CompactionRequest(rqst.getDbName(), rqst.getTableName(),
-              CompactionType.MAJOR);
-      if (rqst.isSetPartNames()) {
-        for (String partName : rqst.getPartNames()) {
-          compactRqst.setPartitionname(partName);
-          compact(compactRqst);
-        }
-      } else {
-        compact(compactRqst);
-      }
-    }
+  @Override
+  @RetrySemantics.Idempotent
+  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
+      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
+              : EventMessage.EventType.BATCH_ACID_WRITE,
+          acidWriteEvent, jdbcResource.getConnection(), sqlGenerator);
   }
 
-  private List<Long> getAbortedWriteIds(ValidWriteIdList validWriteIdList) {
-    return Arrays.stream(validWriteIdList.getInvalidWriteIds())
-        .filter(validWriteIdList::isWriteIdAborted)
-        .boxed()
-        .collect(Collectors.toList());
+  @Override
+  @RetrySemantics.SafeToRetry
+  public void performWriteSetGC() throws MetaException {
+    long commitHighWaterMark = new MinOpenTxnIdWaterMarkFunction(openTxnTimeOutMillis).execute(jdbcResource);
+    jdbcResource.getJdbcTemplate().update(
+        "DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < :hwm",
+        new MapSqlParameterSource()
+            .addValue("hwm", commitHighWaterMark));
   }
 
-  private ValidTxnList getValidTxnList(Connection dbConn, String fullTableName, Long writeId) throws MetaException,
-          SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      String[] names = TxnUtils.getDbTableName(fullTableName);
-      assert names.length == 2;
-      List<String> params = Arrays.asList(names[0], names[1]);
-      String s =
-          "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND "
-              + "\"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = "+ writeId;
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">", quoteString(names[0]),
-            quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        long txnId = rs.getLong(1);
-        return TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), txnId);
-      }
-      throw new MetaException("invalid write id " + writeId + " for table " + fullTableName);
-    } finally {
-      close(rs, pst, null);
-    }
+  @Override
+  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
+    jdbcResource.execute(
+        "UPDATE \"MV_TABLES_USED\" " +
+        "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+ :insertCount" +
+        ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+ :updateCount" +
+        ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+ :deleteCount" +
+        " WHERE \"TBL_ID\"= :tableId",
+        new MapSqlParameterSource()
+            .addValue("insertCount", req.getInsertCount())
+            .addValue("updateCount", req.getUpdatedCount())
+            .addValue("deleteCount", req.getDeletedCount())
+            .addValue("tableId", req.getTableId()), null);
   }
 
+  /**
+   * Get invalidation info for the materialization. Materialization information
+   * contains information about whether there was update/delete/compaction operations on the source
+   * tables used by the materialization since it was created.
+   */
   @Override
   @RetrySemantics.ReadOnly
-  public GetValidWriteIdsResponse getValidWriteIds(GetValidWriteIdsRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ValidTxnList validTxnList;
-
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        // We should prepare the valid write ids list based on validTxnList of current txn.
-        // If no txn exists in the caller, then they would pass null for validTxnList and so it is
-        // required to get the current state of txns to make validTxnList
-        if (rqst.isSetValidTxnList()) {
-          assert rqst.isSetWriteId() == false;
-          validTxnList = new ValidReadTxnList(rqst.getValidTxnList());
-        } else if (rqst.isSetWriteId()) {
-          validTxnList = getValidTxnList(dbConn, rqst.getFullTableNames().get(0), rqst.getWriteId());
-        } else {
-          // Passing 0 for currentTxn means, this validTxnList is not wrt to any txn
-          validTxnList = TxnCommonUtils.createValidReadTxnList(getOpenTxns(dbConn), 0);
-        }
-
-        // Get the valid write id list for all the tables read by the current txn
-        List<TableValidWriteIds> tblValidWriteIdsList = new ArrayList<>();
-        for (String fullTableName : rqst.getFullTableNames()) {
-          tblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, validTxnList));
-        }
-
-        GetValidWriteIdsResponse owr = new GetValidWriteIdsResponse(tblValidWriteIdsList);
-        return owr;
-      } catch (SQLException e) {
-        checkRetryable(e, "getValidWriteIds");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getValidWriteIds(rqst);
-    }
+  public Materialization getMaterializationInvalidationInfo(
+          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
+    return new GetMaterializationInvalidationInfoFunction(creationMetadata, validTxnListStr).execute(jdbcResource);
   }
-  
-  // Method to get the Valid write ids list for the given table
-  // Input fullTableName is expected to be of format <db_name>.<table_name>
-  private TableValidWriteIds getValidWriteIdsForTable(Connection dbConn, String fullTableName,
-                                               ValidTxnList validTxnList) throws SQLException {
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    String[] names = TxnUtils.getDbTableName(fullTableName);
-    assert(names.length == 2);
-    List<String> params = Arrays.asList(names[0], names[1]);
-    try {
-      // Need to initialize to 0 to make sure if nobody modified this table, then current txn
-      // shouldn't read any data.
-      // If there is a conversion from non-acid to acid table, then by default 0 would be assigned as
-      // writeId for data from non-acid table and so writeIdHwm=0 would ensure those data are readable by any txns.
-      long writeIdHwm = 0;
-      List<Long> invalidWriteIdList = new ArrayList<>();
-      long minOpenWriteId = Long.MAX_VALUE;
-      BitSet abortedBits = new BitSet();
-      long txnHwm = validTxnList.getHighWatermark();
-
-      // Find the writeId high water mark based upon txnId high water mark. If found, then, need to
-      // traverse through all write Ids less than writeId HWM to make exceptions list.
-      // The writeHWM = min(NEXT_WRITE_ID.nwi_next-1, max(TXN_TO_WRITE_ID.t2w_writeid under txnHwm))
-      String s = "SELECT MAX(\"T2W_WRITEID\") FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_TXNID\" <= " + txnHwm
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      if (rs.next()) {
-        writeIdHwm = rs.getLong(1);
-      }
-
-      // If no writeIds allocated by txns under txnHwm, then find writeHwm from NEXT_WRITE_ID.
-      if (writeIdHwm <= 0) {
-        // Need to subtract 1 as nwi_next would be the next write id to be allocated but we need highest
-        // allocated write id.
-        s = "SELECT \"NWI_NEXT\"-1 FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-        closeStmt(pst);
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-              quoteString(names[0]), quoteString(names[1]));
-        }
-        rs = pst.executeQuery();
-        if (rs.next()) {
-          writeIdHwm = rs.getLong(1);
-        }
-      }
-      boolean foundValidUncompactedWrite = false;
-      // As writeIdHwm is known, query all writeIds under the writeId HWM.
-      // If any writeId under HWM is allocated by txn > txnId HWM or belongs to open/aborted txns,
-      // then will be added to invalid list. The results should be sorted in ascending order based
-      // on write id. The sorting is needed as exceptions list in ValidWriteIdList would be looked-up
-      // using binary search.
-      s = "SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_WRITEID\" <= " + Long.toString(writeIdHwm)
-              + " AND \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? ORDER BY \"T2W_WRITEID\" ASC";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query<" + s.replace("?", "{}") + ">",
-            quoteString(names[0]), quoteString(names[1]));
-      }
-      rs = pst.executeQuery();
-      while (rs.next()) {
-        long txnId = rs.getLong(1);
-        long writeId = rs.getLong(2);
-        if (validTxnList.isTxnValid(txnId)) {
-          // Skip if the transaction under evaluation is already committed.
-          foundValidUncompactedWrite = true;
-          continue;
-        }
-        // The current txn is either in open or aborted state.
-        // Mark the write ids state as per the txn state.
-        invalidWriteIdList.add(writeId);
-        if (validTxnList.isTxnAborted(txnId)) {
-          abortedBits.set(invalidWriteIdList.size() - 1);
-        } else {
-          minOpenWriteId = Math.min(minOpenWriteId, writeId);
-        }
-      }
-      // If we have compacted writes and some invalid writes on the table,
-      // return the lowest invalid write as a writeIdHwm and set it as invalid.
-      if (!foundValidUncompactedWrite) {
-        long writeId = invalidWriteIdList.isEmpty() ? -1 : invalidWriteIdList.get(0);
-        invalidWriteIdList = new ArrayList<>();
-        abortedBits = new BitSet();
 
-        if (writeId != -1) {
-          invalidWriteIdList.add(writeId);
-          writeIdHwm = writeId;
-          if (writeId != minOpenWriteId) {
-            abortedBits.set(0);
-          }
-        }
-      }
-      ByteBuffer byteBuffer = ByteBuffer.wrap(abortedBits.toByteArray());
-      TableValidWriteIds owi = new TableValidWriteIds(fullTableName, writeIdHwm, invalidWriteIdList, byteBuffer);
-      if (minOpenWriteId < Long.MAX_VALUE) {
-        owi.setMinOpenWriteId(minOpenWriteId);
-      }
-      return owi;
-    } finally {
-      closeStmt(pst);
-      close(rs);
-    }
+  @Override
+  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    return new LockMaterializationRebuildFunction(dbName, tableName, txnId, mutexAPI).execute(jdbcResource);
   }
 
   @Override
-  @RetrySemantics.Idempotent
-  public AllocateTableWriteIdsResponse allocateTableWriteIds(AllocateTableWriteIdsRequest rqst)
-          throws MetaException {
-    List<Long> txnIds;
-    String dbName = rqst.getDbName().toLowerCase();
-    String tblName = rqst.getTableName().toLowerCase();
-    boolean shouldReallocate = rqst.isReallocate();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      List<TxnToWriteId> txnToWriteIds = new ArrayList<>();
-      List<TxnToWriteId> srcTxnToWriteIds = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
+  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId) throws MetaException {
+    int result = jdbcResource.execute(
+        "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
+            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
+            " WHERE \"MRL_TXN_ID\" = " + txnId +
+            " AND \"MRL_DB_NAME\" = ?" +
+            " AND \"MRL_TBL_NAME\" = ?",
+        new MapSqlParameterSource()
+            .addValue("now", Instant.now().toEpochMilli())
+            .addValue("txnId", txnId)
+            .addValue("dbName", dbName)
+            .addValue("tableNane", tableName),
+        ParameterizedCommand.AT_LEAST_ONE_ROW);
+    return result >= 1;
+  }
 
-        if (rqst.isSetReplPolicy()) {
-          srcTxnToWriteIds = rqst.getSrcTxnToWriteIdList();
-          List<Long> srcTxnIds = new ArrayList<>();
-          assert (rqst.isSetSrcTxnToWriteIdList());
-          assert (!rqst.isSetTxnIds());
-          assert (!srcTxnToWriteIds.isEmpty());
-
-          for (TxnToWriteId txnToWriteId : srcTxnToWriteIds) {
-            srcTxnIds.add(txnToWriteId.getTxnId());
-          }
-          txnIds = getTargetTxnIdList(rqst.getReplPolicy(), srcTxnIds, dbConn);
-          if (srcTxnIds.size() != txnIds.size()) {
-            // Idempotent case where txn was already closed but gets allocate write id event.
-            // So, just ignore it and return empty list.
-            LOG.info("Idempotent case: Target txn id is missing for source txn id : {} and repl policy {}", srcTxnIds,
-                rqst.getReplPolicy());
-            return new AllocateTableWriteIdsResponse(txnToWriteIds);
-          }
-        } else {
-          assert (!rqst.isSetSrcTxnToWriteIdList());
-          assert (rqst.isSetTxnIds());
-          txnIds = rqst.getTxnIds();
-        }
-
-        //Easiest check since we can't differentiate do we handle singleton list or list with multiple txn ids.
-        if (txnIds.size() > 1) {
-          Collections.sort(txnIds); //easier to read logs and for assumption done in replication flow
-        }
-
-        // Check if all the input txns are in valid state.
-        // Write IDs should be allocated only for open and not read-only transactions.
-        try (Statement stmt = dbConn.createStatement()) {
-          if (!isTxnsOpenAndNotReadOnly(txnIds, stmt)) {
-            String errorMsg = "Write ID allocation on " + TableName.getDbTable(dbName, tblName)
-                    + " failed for input txns: "
-                    + getAbortedAndReadOnlyTxns(txnIds, stmt)
-                    + getCommittedTxns(txnIds, stmt);
-            LOG.error(errorMsg);
-
-            throw new IllegalStateException("Write ID allocation failed on " + TableName.getDbTable(dbName, tblName)
-                    + " as not all input txns in open state or read-only");
-          }
-        }
-
-        List<String> queries = new ArrayList<>();
-        StringBuilder prefix = new StringBuilder();
-        StringBuilder suffix = new StringBuilder();
-        long writeId;
-        int allocatedTxnsCount = 0;
-        List<String> params = Arrays.asList(dbName, tblName);
-        if (shouldReallocate) {
-          // during query recompilation after lock acquistion, it is important to realloc new writeIds
-          // to ensure writeIds are committed in increasing order.
-          prefix.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            int numRowsDeleted = pStmt.executeUpdate();
-            LOG.info("Removed {} prior writeIds during reallocation", numRowsDeleted);
-            closeStmt(pStmt);
-          }
-        } else {
-          // Traverse the TXN_TO_WRITE_ID to see if any of the input txns already have allocated a
-          // write id for the same db.table. If yes, then need to reuse it else have to allocate new one
-          // The write id would have been already allocated in case of multi-statement txns where
-          // first write on a table will allocate write id and rest of the writes should re-use it.
-          prefix.append("SELECT \"T2W_TXNID\", \"T2W_WRITEID\" FROM \"TXN_TO_WRITE_ID\" WHERE")
-                .append(" \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND ");
-          TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
-              txnIds, "\"T2W_TXNID\"", false, false);
-          for (String query : queries) {
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            rs = pStmt.executeQuery();
-            while (rs.next()) {
-              // If table write ID is already allocated for the given transaction, then just use it
-              long txnId = rs.getLong(1);
-              writeId = rs.getLong(2);
-              txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-              allocatedTxnsCount++;
-              LOG.info("Reused already allocated writeID: {} for txnId: {}", writeId, txnId);
-            }
-            closeStmt(pStmt);
-          }
-        }
-
-        // Batch allocation should always happen atomically. Either write ids for all txns is allocated or none.
-        long numOfWriteIds = txnIds.size();
-        assert ((allocatedTxnsCount == 0) || (numOfWriteIds == allocatedTxnsCount));
-        if (allocatedTxnsCount == numOfWriteIds) {
-          // If all the txns in the list have pre-allocated write ids for the given table, then just return.
-          // This is for idempotent case.
-          return new AllocateTableWriteIdsResponse(txnToWriteIds);
-        }
-
-        long srcWriteId = 0;
-        if (rqst.isSetReplPolicy()) {
-          // In replication flow, we always need to allocate write ID equal to that of source.
-          assert (srcTxnToWriteIds != null);
-          srcWriteId = srcTxnToWriteIds.get(0).getWriteId();
-        }
-
-
-        // There are some txns in the list which does not have write id allocated and hence go ahead and do it.
-        // Get the next write id for the given table and update it with new next write id.
-        // This is select for update query which takes a lock if the table entry is already there in NEXT_WRITE_ID
-        String s = sqlGenerator.addForUpdateClause(
-            "SELECT \"NWI_NEXT\" FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?");
-        closeStmt(pStmt);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        rs = pStmt.executeQuery();
-        if (!rs.next()) {
-          // First allocation of write id should add the table to the next_write_id meta table
-          // The initial value for write id should be 1 and hence we add 1 with number of write ids allocated here
-          // For repl flow, we need to force set the incoming write id.
-          writeId = (srcWriteId > 0) ? srcWriteId : 1;
-          s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                  + (writeId + numOfWriteIds) + ")";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.execute();
-        } else {
-          long nextWriteId = rs.getLong(1);
-          writeId = (srcWriteId > 0) ? srcWriteId : nextWriteId;
-
-          // Update the NEXT_WRITE_ID for the given table after incrementing by number of write ids allocated
-          s = "UPDATE \"NEXT_WRITE_ID\" SET \"NWI_NEXT\" = " + (writeId + numOfWriteIds)
-                  + " WHERE \"NWI_DATABASE\" = ? AND \"NWI_TABLE\" = ?";
-          closeStmt(pStmt);
-          pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-          if (LOG.isDebugEnabled()) {
-            LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-                quoteString(dbName), quoteString(tblName));
-          }
-          pStmt.executeUpdate();
-
-          // For repl flow, if the source write id is mismatching with target next write id, then current
-          // metadata in TXN_TO_WRITE_ID is stale for this table and hence need to clean-up TXN_TO_WRITE_ID.
-          // This is possible in case of first incremental repl after bootstrap where concurrent write
-          // and drop table was performed at source during bootstrap dump.
-          if ((srcWriteId > 0) && (srcWriteId != nextWriteId)) {
-            s = "DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ?";
-            closeStmt(pStmt);
-            pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("Going to execute delete <" + s.replace("?", "{}") + ">",
-                  quoteString(dbName), quoteString(tblName));
-            }
-            pStmt.executeUpdate();
-          }
-        }
-
-        // Map the newly allocated write ids against the list of txns which doesn't have pre-allocated write ids
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_TO_WRITE_ID_INSERT_QUERY)) {
-          for (long txnId : txnIds) {
-            pstmt.setLong(1, txnId);
-            pstmt.setString(2, dbName);
-            pstmt.setString(3, tblName);
-            pstmt.setLong(4, writeId);
-            pstmt.addBatch();
-
-            txnToWriteIds.add(new TxnToWriteId(txnId, writeId));
-            LOG.info("Allocated writeId: {} for txnId: {}", writeId, txnId);
-            writeId++;
-            if (txnToWriteIds.size() % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                  maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (txnToWriteIds.size() % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_TO_WRITE_ID_INSERT_QUERY,
-                txnToWriteIds.size() % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-
-        if (transactionalListeners != null) {
-          MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-                  EventMessage.EventType.ALLOC_WRITE_ID,
-                  new AllocWriteIdEvent(txnToWriteIds, dbName, tblName),
-                  dbConn, sqlGenerator);
-        }
-
-        LOG.info("Allocated write ids for dbName={}, tblName={} (txnIds: {})", dbName, tblName, rqst.getTxnIds());
-        dbConn.commit();
-        return new AllocateTableWriteIdsResponse(txnToWriteIds);
-      } catch (SQLException e) {
-        LOG.error("Exception during write ids allocation for request={}. Will retry if possible.", rqst, e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "allocateTableWriteIds(" + rqst + ")", true);
-        throw new MetaException("Unable to update transaction database "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return allocateTableWriteIds(rqst);
-    }
-  }
-
-  @Override
-  public MaxAllocatedTableWriteIdResponse getMaxAllocatedTableWrited(MaxAllocatedTableWriteIdRequest rqst) throws MetaException {
-    String dbName = rqst.getDbName();
-    String tableName = rqst.getTableName();
-    try {
-      Connection dbConn = null;
-      PreparedStatement pStmt = null;
-      ResultSet rs = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        pStmt = sqlGenerator.prepareStmtWithParameters(dbConn, SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID,
-            Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + SELECT_NWI_NEXT_FROM_NEXT_WRITE_ID.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        rs = pStmt.executeQuery();
-        // If there is no record, we never allocated anything
-        long maxWriteId = 0l;
-        if (rs.next()) {
-          // The row contains the nextId not the previously allocated
-          maxWriteId = rs.getLong(1) - 1;
-        }
-        return new MaxAllocatedTableWriteIdResponse(maxWriteId);
-      } catch (SQLException e) {
-        LOG.error(
-            "Exception during reading the max allocated writeId for dbName={}, tableName={}. Will retry if possible.",
-            dbName, tableName, e);
-        checkRetryable(e, "getMaxAllocatedTableWrited(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(rs, pStmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return getMaxAllocatedTableWrited(rqst);
-    }
-  }
-
-  @Override
-  public void seedWriteId(SeedTableWriteIdsRequest rqst)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        //since this is on conversion from non-acid to acid, NEXT_WRITE_ID should not have an entry
-        //for this table.  It also has a unique index in case 'should not' is violated
-
-        // First allocation of write id should add the table to the next_write_id meta table
-        // The initial value for write id should be 1 and hence we add 1 with number of write ids
-        // allocated here
-        String s = "INSERT INTO \"NEXT_WRITE_ID\" (\"NWI_DATABASE\", \"NWI_TABLE\", \"NWI_NEXT\") VALUES (?, ?, "
-                + Long.toString(rqst.getSeedWriteId() + 1) + ")";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(rqst.getDbName(), rqst.getTableName()));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute insert <" + s.replace("?", "{}") + ">",
-              quoteString(rqst.getDbName()), quoteString(rqst.getTableName()));
-        }
-        pst.execute();
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedWriteId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      seedWriteId(rqst);
-    }
-  }
-
-  @Override
-  public void seedTxnId(SeedTxnIdRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        /*
-         * Locking the txnLock an exclusive way, we do not want to set the txnId backward accidentally
-         * if there are concurrent open transactions
-         */
-        acquireTxnLock(stmt, false);
-        long highWaterMark = getHighWaterMark(stmt);
-        if (highWaterMark >= rqst.getSeedTxnId()) {
-          throw new MetaException(MessageFormat
-              .format("Invalid txnId seed {}, the highWaterMark is {}", rqst.getSeedTxnId(), highWaterMark));
-        }
-        TxnUtils.seedTxnSequence(dbConn, conf, stmt, rqst.getSeedTxnId());
-        dbConn.commit();
-
-      } catch (SQLException e) {
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "seedTxnId(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " + StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      seedTxnId(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void addWriteNotificationLog(ListenerEvent acidWriteEvent) throws MetaException {
-      Connection dbConn = jdbcResource.getConnection();
-      MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
-          acidWriteEvent instanceof AcidWriteEvent ? EventMessage.EventType.ACID_WRITE
-              : EventMessage.EventType.BATCH_ACID_WRITE,
-          acidWriteEvent, dbConn, sqlGenerator);
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void performWriteSetGC() throws MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      long commitHighWaterMark = getMinOpenTxnIdWaterMark(dbConn);
-      int delCnt = stmt.executeUpdate("DELETE FROM \"WRITE_SET\" WHERE \"WS_COMMIT_ID\" < " + commitHighWaterMark);
-      LOG.info("Deleted {} obsolete rows from WRITE_SET", delCnt);
-      dbConn.commit();
-    } catch (SQLException ex) {
-      LOG.warn("WriteSet GC failed due to " + getMessage(ex), ex);
-    } finally {
-      close(null, stmt, dbConn);
-    }
-  }
-
-  protected long getMinOpenTxnIdWaterMark(Connection dbConn) throws SQLException, MetaException {
-    /**
-     * We try to find the highest transactionId below everything was committed or aborted.
-     * For that we look for the lowest open transaction in the TXNS and the TxnMinTimeout boundary,
-     * because it is guaranteed there won't be open transactions below that.
-     */
-    long minOpenTxn;
-    try (Statement stmt = dbConn.createStatement()) {
-      try (ResultSet rs = stmt
-          .executeQuery("SELECT MIN(\"TXN_ID\") FROM \"TXNS\" WHERE \"TXN_STATE\"=" + TxnStatus.OPEN)) {
-        if (!rs.next()) {
-          throw new IllegalStateException("Scalar query returned no rows?!?!!");
-        }
-        minOpenTxn = rs.getLong(1);
-        if (rs.wasNull()) {
-          minOpenTxn = Long.MAX_VALUE;
-        }
-      }
-    } catch (SQLException e) {
-      throw new UncategorizedSQLException(null, null, e);
-    }
-    long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-    LOG.debug("MinOpenTxnIdWaterMark calculated with minOpenTxn {}, lowWaterMark {}", minOpenTxn, lowWaterMark);
-    return Long.min(minOpenTxn, lowWaterMark + 1);
-  }
-
-  @Override
-  public void updateTransactionStatistics(UpdateTransactionalStatsRequest req) throws MetaException {
-    String queryText = "UPDATE \"MV_TABLES_USED\" " +
-            "SET \"INSERTED_COUNT\"=\"INSERTED_COUNT\"+?" +
-            ",\"UPDATED_COUNT\"=\"UPDATED_COUNT\"+?" +
-            ",\"DELETED_COUNT\"=\"DELETED_COUNT\"+?" +
-            " WHERE \"TBL_ID\"=?";
-    try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-      try (PreparedStatement pstmt = dbConn.prepareStatement(queryText)) {
-        pstmt.setLong(1, req.getInsertCount());
-        pstmt.setLong(2, req.getUpdatedCount());
-        pstmt.setLong(3, req.getDeletedCount());
-        pstmt.setLong(4, req.getTableId());
-        LOG.debug("Going to execute query <{}>", queryText);
-        int res = pstmt.executeUpdate();
-        dbConn.commit();
-        LOG.debug("Updated {} records tblId={}", res, req.getTableId());
-      }
-    } catch (SQLException ex) {
-      LOG.warn("Unable to update transactional statistics tblId=" + req.getTableId(), ex);
-      throw new MetaException("Unable to update transactional statistics" + " " + StringUtils.stringifyException(ex));
-    }
-  }
-
-  /**
-   * Get invalidation info for the materialization. Materialization information
-   * contains information about whether there was update/delete/compaction operations on the source
-   * tables used by the materialization since it was created.
-   */
-  @Override
-  @RetrySemantics.ReadOnly
-  public Materialization getMaterializationInvalidationInfo(
-          CreationMetadata creationMetadata, String validTxnListStr) throws MetaException {
-    if (creationMetadata.getTablesUsed().isEmpty()) {
-      // Bail out
-      LOG.warn("Materialization creation metadata does not contain any table");
-      return null;
-    }
-
-    // We are composing a query that returns a single row if an update happened after
-    // the materialization was created. Otherwise, query returns 0 rows.
-
-    // Parse validReaderWriteIdList from creation metadata
-    MaterializationSnapshot mvSnapshot = MaterializationSnapshot.fromJson(creationMetadata.getValidTxnList());
-    if (mvSnapshot.getTableSnapshots() != null && !mvSnapshot.getTableSnapshots().isEmpty()) {
-      // Incremental rebuild of MVs on Iceberg sources is not supported.
-      return null;
-    }
-    final ValidTxnWriteIdList validReaderWriteIdList = new ValidTxnWriteIdList(mvSnapshot.getValidTxnList());
-
-    // Parse validTxnList
-    final ValidReadTxnList currentValidTxnList = new ValidReadTxnList(validTxnListStr);
-    // Get the valid write id list for the tables in current state
-    final List<TableValidWriteIds> currentTblValidWriteIdsList = new ArrayList<>();
-    Connection dbConn = null;
-    for (String fullTableName : creationMetadata.getTablesUsed()) {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        currentTblValidWriteIdsList.add(getValidWriteIdsForTable(dbConn, fullTableName, currentValidTxnList));
-      } catch (SQLException ex) {
-        String errorMsg = "Unable to query Valid writeIds of table " + fullTableName;
-        LOG.warn(errorMsg, ex);
-        throw new MetaException(errorMsg + " " + StringUtils.stringifyException(ex));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    }
-    final ValidTxnWriteIdList currentValidReaderWriteIdList = TxnCommonUtils.createValidTxnWriteIdList(
-            currentValidTxnList.getHighWatermark(), currentTblValidWriteIdsList);
-
-    List<String> params = new ArrayList<>();
-    StringBuilder queryUpdateDelete = new StringBuilder();
-    StringBuilder queryCompletedCompactions = new StringBuilder();
-    StringBuilder queryCompactionQueue = new StringBuilder();
-    // compose a query that select transactions containing an update...
-    queryUpdateDelete.append("SELECT \"CTC_UPDATE_DELETE\" FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_UPDATE_DELETE\" ='Y' AND (");
-    queryCompletedCompactions.append("SELECT 1 FROM \"COMPLETED_COMPACTIONS\" WHERE (");
-    queryCompactionQueue.append("SELECT 1 FROM \"COMPACTION_QUEUE\" WHERE (");
-    int i = 0;
-    for (String fullyQualifiedName : creationMetadata.getTablesUsed()) {
-      ValidWriteIdList tblValidWriteIdList =
-              validReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (tblValidWriteIdList == null) {
-        LOG.warn("ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-
-      // First, we check whether the low watermark has moved for any of the tables.
-      // If it has, we return true, since it is not incrementally refreshable, e.g.,
-      // one of the commits that are not available may be an update/delete.
-      ValidWriteIdList currentTblValidWriteIdList =
-              currentValidReaderWriteIdList.getTableValidWriteIdList(fullyQualifiedName);
-      if (currentTblValidWriteIdList == null) {
-        LOG.warn("Current ValidWriteIdList for table {} not present in creation metadata, this should not happen", fullyQualifiedName);
-        return null;
-      }
-      if (!Objects.equals(currentTblValidWriteIdList.getMinOpenWriteId(), tblValidWriteIdList.getMinOpenWriteId())) {
-        LOG.debug("Minimum open write id do not match for table {}", fullyQualifiedName);
-        return null;
-      }
-
-      // ...for each of the tables that are part of the materialized view,
-      // where the transaction had to be committed after the materialization was created...
-      if (i != 0) {
-        queryUpdateDelete.append("OR");
-        queryCompletedCompactions.append("OR");
-        queryCompactionQueue.append("OR");
-      }
-      String[] names = TxnUtils.getDbTableName(fullyQualifiedName);
-      assert (names.length == 2);
-      queryUpdateDelete.append(" (\"CTC_DATABASE\"=? AND \"CTC_TABLE\"=?");
-      queryCompletedCompactions.append(" (\"CC_DATABASE\"=? AND \"CC_TABLE\"=?");
-      queryCompactionQueue.append(" (\"CQ_DATABASE\"=? AND \"CQ_TABLE\"=?");
-      params.add(names[0]);
-      params.add(names[1]);
-      queryUpdateDelete.append(" AND (\"CTC_WRITEID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryCompletedCompactions.append(" AND (\"CC_HIGHEST_WRITE_ID\" > " + tblValidWriteIdList.getHighWatermark());
-      queryUpdateDelete.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CTC_WRITEID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryCompletedCompactions.append(tblValidWriteIdList.getInvalidWriteIds().length == 0 ? ") " :
-              " OR \"CC_HIGHEST_WRITE_ID\" IN(" + StringUtils.join(",",
-                      Arrays.asList(ArrayUtils.toObject(tblValidWriteIdList.getInvalidWriteIds()))) + ") ) ");
-      queryUpdateDelete.append(") ");
-      queryCompletedCompactions.append(") ");
-      queryCompactionQueue.append(") ");
-      i++;
-    }
-    // ... and where the transaction has already been committed as per snapshot taken
-    // when we are running current query
-    queryUpdateDelete.append(") AND \"CTC_TXNID\" <= " + currentValidTxnList.getHighWatermark());
-    queryUpdateDelete.append(currentValidTxnList.getInvalidTransactions().length == 0 ? " " :
-            " AND \"CTC_TXNID\" NOT IN(" + StringUtils.join(",",
-                    Arrays.asList(ArrayUtils.toObject(currentValidTxnList.getInvalidTransactions()))) + ") ");
-    queryCompletedCompactions.append(")");
-    queryCompactionQueue.append(") ");
-
-    boolean hasUpdateDelete = executeBoolean(queryUpdateDelete.toString(), params,
-            "Unable to retrieve materialization invalidation information: completed transaction components.");
-
-    // Execute query
-    queryCompletedCompactions.append(" UNION ");
-    queryCompletedCompactions.append(queryCompactionQueue.toString());
-    List<String> paramsTwice = new ArrayList<>(params);
-    paramsTwice.addAll(params);
-    boolean hasCompaction = executeBoolean(queryCompletedCompactions.toString(), paramsTwice,
-            "Unable to retrieve materialization invalidation information: compactions");
-
-    return new Materialization(hasUpdateDelete, hasCompaction);
-  }
-
-  private boolean executeBoolean(String queryText, List<String> params, String errorMessage) throws MetaException {
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      LOG.debug("Going to execute query <{}>", queryText);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, queryText, params);
-      pst.setMaxRows(1);
-      rs = pst.executeQuery();
-
-      return rs.next();
-    } catch (SQLException ex) {
-      LOG.warn(errorMessage, ex);
-      throw new MetaException(errorMessage + " " + StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-    }
-  }
-
-  @Override
-  public LockResponse lockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-
-    if (LOG.isDebugEnabled()) {
-      LOG.debug("Acquiring lock for materialization rebuild with {} for {}",
-          JavaUtils.txnIdToString(txnId), TableName.getDbTable(dbName, tableName));
-    }
-
-    TxnStore.MutexAPI.LockHandle handle = null;
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      lockInternal();
-      /**
-       * MUTEX_KEY.MaterializationRebuild lock ensures that there is only 1 entry in
-       * Initiated/Working state for any resource. This ensures we do not run concurrent
-       * rebuild operations on any materialization.
-       */
-      handle = getMutexAPI().acquireLock(MUTEX_KEY.MaterializationRebuild.name());
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-      List<String> params = Arrays.asList(dbName, tableName);
-      String selectQ = "SELECT \"MRL_TXN_ID\" FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-          " \"MRL_DB_NAME\" = ? AND \"MRL_TBL_NAME\" = ?";
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, selectQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute query <" + selectQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      rs = pst.executeQuery();
-      if(rs.next()) {
-        LOG.info("Ignoring request to rebuild {}/{} since it is already being rebuilt", dbName, tableName);
-        return new LockResponse(txnId, LockState.NOT_ACQUIRED);
-      }
-      String insertQ = "INSERT INTO \"MATERIALIZATION_REBUILD_LOCKS\" " +
-          "(\"MRL_TXN_ID\", \"MRL_DB_NAME\", \"MRL_TBL_NAME\", \"MRL_LAST_HEARTBEAT\") VALUES (" + txnId +
-          ", ?, ?, " + Instant.now().toEpochMilli() + ")";
-      closeStmt(pst);
-      pst = sqlGenerator.prepareStmtWithParameters(dbConn, insertQ, params);
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Going to execute update <" + insertQ.replace("?", "{}") + ">",
-            quoteString(dbName), quoteString(tableName));
-      }
-      pst.executeUpdate();
-      LOG.debug("Going to commit");
-      dbConn.commit();
-      return new LockResponse(txnId, LockState.ACQUIRED);
-    } catch (SQLException ex) {
-      LOG.warn("lockMaterializationRebuild failed due to " + getMessage(ex), ex);
-      throw new MetaException("Unable to retrieve materialization invalidation information due to " +
-          StringUtils.stringifyException(ex));
-    } finally {
-      close(rs, pst, dbConn);
-      if(handle != null) {
-        handle.releaseLocks();
-      }
-      unlockInternal();
-    }
-  }
-
-  @Override
-  public boolean heartbeatLockMaterializationRebuild(String dbName, String tableName, long txnId)
-      throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        String s = "UPDATE \"MATERIALIZATION_REBUILD_LOCKS\"" +
-            " SET \"MRL_LAST_HEARTBEAT\" = " + Instant.now().toEpochMilli() +
-            " WHERE \"MRL_TXN_ID\" = " + txnId +
-            " AND \"MRL_DB_NAME\" = ?" +
-            " AND \"MRL_TBL_NAME\" = ?";
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, Arrays.asList(dbName, tableName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute update <" + s.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tableName));
-        }
-        int rc = pst.executeUpdate();
-        if (rc < 1) {
-          LOG.debug("Going to rollback");
-          dbConn.rollback();
-          LOG.info("No lock found for rebuild of {} when trying to heartbeat", TableName.getDbTable(dbName, tableName));
-          // It could not be renewed, return that information
-          return false;
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        // It could be renewed, return that information
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e,
-            "heartbeatLockMaterializationRebuild(" + TableName.getDbTable(dbName, tableName) + ", " + txnId + ")");
-        throw new MetaException("Unable to heartbeat rebuild lock due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return heartbeatLockMaterializationRebuild(dbName, tableName ,txnId);
-    }
-  }
-
-  @Override
-  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
-    try {
-      // Aux values
-      long cnt = 0L;
-      List<Long> txnIds = new ArrayList<>();
-      long timeoutTime = Instant.now().toEpochMilli() - timeout;
-
-      Connection dbConn = null;
-      Statement stmt = null;
-      ResultSet rs = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-
-        String selectQ = "SELECT \"MRL_TXN_ID\", \"MRL_LAST_HEARTBEAT\" FROM \"MATERIALIZATION_REBUILD_LOCKS\"";
-        LOG.debug("Going to execute query <{}>", selectQ);
-        rs = stmt.executeQuery(selectQ);
-        while(rs.next()) {
-          long lastHeartbeat = rs.getLong(2);
-          if (lastHeartbeat < timeoutTime) {
-            // The heartbeat has timeout, double check whether we can remove it
-            long txnId = rs.getLong(1);
-            if (validTxnList.isTxnValid(txnId) || validTxnList.isTxnAborted(txnId)) {
-              // Txn was committed (but notification was not received) or it was aborted.
-              // Either case, we can clean it up
-              txnIds.add(txnId);
-            }
-          }
-        }
-        if (!txnIds.isEmpty()) {
-          String deleteQ = "DELETE FROM \"MATERIALIZATION_REBUILD_LOCKS\" WHERE" +
-              " \"MRL_TXN_ID\" IN(" + StringUtils.join(",", txnIds) + ") ";
-          LOG.debug("Going to execute update <{}>", deleteQ);
-          cnt = stmt.executeUpdate(deleteQ);
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return cnt;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupMaterializationRebuildLocks");
-        throw new MetaException("Unable to clean rebuild locks due to " +
-            StringUtils.stringifyException(e));
-      } finally {
-        close(rs, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return cleanupMaterializationRebuildLocks(validTxnList, timeout);
-    }
-  }
+  @Override
+  public long cleanupMaterializationRebuildLocks(ValidTxnList validTxnList, long timeout) throws MetaException {
+    return new ReleaseMaterializationRebuildLocks(validTxnList, timeout).execute(jdbcResource);
+  }
 
   /**
    * As much as possible (i.e. in absence of retries) we want both operations to be done on the same
    * connection (but separate transactions).
-   *
    * Retry-by-caller note: If the call to lock is from a transaction, then in the worst case
-   * there will be a duplicate set of locks but both sets will belong to the same txn so they
-   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
-   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
-   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
-   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
-   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
-   * set of locks times out.
-   */
-  @RetrySemantics.CannotRetry
-  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
-    ConnectionLockIdPair connAndLockId = enqueueLockWithRetry(rqst);
-    try {
-      return checkLockWithRetry(connAndLockId.dbConn, connAndLockId.extLockId, rqst.getTxnid(),
-          rqst.isZeroWaitReadEnabled(), rqst.isExclusiveCTAS());
-    }
-    catch(NoSuchLockException e) {
-      // This should never happen, as we just added the lock id
-      throw new MetaException("Couldn't find a lock we just created! " + e.getMessage());
-    }
-  }
-  private static final class ConnectionLockIdPair {
-    private final Connection dbConn;
-    private final long extLockId;
-    private ConnectionLockIdPair(Connection dbConn, long extLockId) {
-      this.dbConn = dbConn;
-      this.extLockId = extLockId;
-    }
-  }
-
-  /**
-   * Note that by definition select for update is divorced from update, i.e. you executeQuery() to read
-   * and then executeUpdate().  One other alternative would be to actually update the row in TXNS but
-   * to the same value as before thus forcing db to acquire write lock for duration of the transaction.
-   *
-   * SELECT ... FOR UPDATE locks the row until the transaction commits or rolls back.
-   * Second connection using `SELECT ... FOR UPDATE` will suspend until the lock is released.
-   * @return the txnType wrapped in an {@link Optional}
-   * @throws SQLException
-   * @throws MetaException
-   */
-  private TxnType getOpenTxnTypeAndLock(Statement stmt, long txnId) throws SQLException, MetaException {
-    String query = "SELECT \"TXN_TYPE\" FROM \"TXNS\" WHERE \"TXN_ID\" = " + txnId
-        + " AND \"TXN_STATE\" = " + TxnStatus.OPEN;
-    try (ResultSet rs = stmt.executeQuery(sqlGenerator.addForUpdateClause(query))) {
-      return rs.next() ? TxnType.findByValue(rs.getInt(1)) : null;
-    }
-  }
-
-  /**
-   * This enters locks into the queue in {@link #LOCK_WAITING} mode.
-   *
-   * Isolation Level Notes:
-   * 1. We use S4U (withe read_committed) to generate the next (ext) lock id.  This serializes
-   * any 2 {@code enqueueLockWithRetry()} calls.
-   * 2. We use S4U on the relevant TXNS row to block any concurrent abort/commit/etc operations
-   * @see #checkLockWithRetry(Connection, long, long, boolean, boolean)
-   */
-  private ConnectionLockIdPair enqueueLockWithRetry(LockRequest rqst)
-      throws NoSuchTxnException, TxnAbortedException, MetaException {
-    boolean success = false;
-    Connection dbConn = null;
-    try {
-      Statement stmt = null;
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        long txnid = rqst.getTxnid();
-        stmt = dbConn.createStatement();
-        if (isValidTxn(txnid)) {
-          //this also ensures that txn is still there in expected state
-          TxnType txnType = getOpenTxnTypeAndLock(stmt, txnid);
-          if (txnType == null) {
-            ensureValidTxn(dbConn, txnid, stmt);
-            shouldNeverHappen(txnid);
-          }
-        }
-        /* Insert txn components and hive locks (with a temp extLockId) first, before getting the next lock ID in a select-for-update.
-           This should minimize the scope of the S4U and decrease the table lock duration. */
-        insertTxnComponents(txnid, rqst, dbConn);
-        long tempExtLockId = insertHiveLocksWithTemporaryExtLockId(txnid, dbConn, rqst);
-
-        /** Get the next lock id.
-         * This has to be atomic with adding entries to HIVE_LOCK entries (1st add in W state) to prevent a race.
-         * Suppose ID gen is a separate txn and 2 concurrent lock() methods are running.  1st one generates nl_next=7,
-         * 2nd nl_next=8.  Then 8 goes first to insert into HIVE_LOCKS and acquires the locks.  Then 7 unblocks,
-         * and add it's W locks but it won't see locks from 8 since to be 'fair' {@link #checkLock(java.sql.Connection, long)}
-         * doesn't block on locks acquired later than one it's checking*/
-        long extLockId = getNextLockIdForUpdate(dbConn, stmt);
-        incrementLockIdAndUpdateHiveLocks(stmt, extLockId, tempExtLockId);
-
-        dbConn.commit();
-        success = true;
-        return new ConnectionLockIdPair(dbConn, extLockId);
-      } catch (SQLException e) {
-        LOG.error("enqueueLock failed for request: {}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "enqueueLockWithRetry(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        if (!success) {
-          /* This needs to return a "live" connection to be used by operation that follows it.
-          Thus it only closes Connection on failure/retry. */
-          closeDbConn(dbConn);
-        }
-        unlockInternal();
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry enqueueLock for request: {}, after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return enqueueLockWithRetry(rqst);
-    }
-  }
-
-  private long getNextLockIdForUpdate(Connection dbConn, Statement stmt) throws SQLException, MetaException {
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NL_NEXT\" FROM \"NEXT_LOCK_ID\"");
-    LOG.debug("Going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        LOG.error("Failure to get next lock ID for update! SELECT query returned empty ResultSet.");
-        dbConn.rollback();
-        throw new MetaException("Transaction tables not properly " +
-                "initialized, no record found in next_lock_id");
-      }
-      return rs.getLong(1);
-    }
-  }
-
-  private void incrementLockIdAndUpdateHiveLocks(Statement stmt, long extLockId, long tempId) throws SQLException {
-    String incrCmd = String.format(INCREMENT_NEXT_LOCK_ID_QUERY, (extLockId + 1));
-    // update hive locks entries with the real EXT_LOCK_ID (replace temp ID)
-    String updateLocksCmd = String.format(UPDATE_HIVE_LOCKS_EXT_ID_QUERY, extLockId, tempId);
-    LOG.debug("Going to execute updates in batch: <{}>, and <{}>", incrCmd, updateLocksCmd);
-    stmt.addBatch(incrCmd);
-    stmt.addBatch(updateLocksCmd);
-    stmt.executeBatch();
-  }
-
-  private void insertTxnComponents(long txnid, LockRequest rqst, Connection dbConn) throws SQLException {
-    if (txnid > 0) {
-      Map<Pair<String, String>, Optional<Long>> writeIdCache = new HashMap<>();
-      try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-        // For each component in this lock request,
-        // add an entry to the txn_components table
-        int insertCounter = 0;
-
-        Predicate<LockComponent> isDynPart = lc -> lc.isSetIsDynamicPartitionWrite() && lc.isIsDynamicPartitionWrite();
-        Function<LockComponent, Pair<String, String>> groupKey = lc ->
-            Pair.of(normalizeCase(lc.getDbname()), normalizeCase(lc.getTablename()));
-
-        Set<Pair<String, String>> isDynPartUpdate = rqst.getComponent().stream().filter(isDynPart)
-          .filter(lc -> lc.getOperationType() == DataOperationType.UPDATE || lc.getOperationType() == DataOperationType.DELETE)
-          .map(groupKey)
-        .collect(Collectors.toSet());
-
-        for (LockComponent lc : rqst.getComponent()) {
-          if (lc.isSetIsTransactional() && !lc.isIsTransactional()) {
-            //we don't prevent using non-acid resources in a txn but we do lock them
-            continue;
-          }
-          if (!shouldUpdateTxnComponent(txnid, rqst, lc)) {
-            continue;
-          }
-          String dbName = normalizeCase(lc.getDbname());
-          String tblName = normalizeCase(lc.getTablename());
-          String partName = normalizePartitionCase(lc.getPartitionname());
-          OperationType opType = OperationType.fromDataOperationType(lc.getOperationType());
-
-          if (isDynPart.test(lc)) {
-            partName = null;
-            if (writeIdCache.containsKey(groupKey.apply(lc))) {
-              continue;
-            }
-            opType = isDynPartUpdate.contains(groupKey.apply(lc)) ? OperationType.UPDATE : OperationType.INSERT;
-          }
-          Optional<Long> writeId = getWriteId(writeIdCache, dbName, tblName, txnid, dbConn);
-
-          pstmt.setLong(1, txnid);
-          pstmt.setString(2, dbName);
-          pstmt.setString(3, tblName);
-          pstmt.setString(4, partName);
-          pstmt.setString(5, opType.getSqlConst());
-          pstmt.setObject(6, writeId.orElse(null));
-
-          pstmt.addBatch();
-          insertCounter++;
-          if (insertCounter % maxBatchSize == 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        if (insertCounter % maxBatchSize != 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-              insertCounter % maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-    }
-  }
-
-  private Optional<Long> getWriteId(Map<Pair<String, String>, Optional<Long>> writeIdCache, String dbName, String tblName, long txnid, Connection dbConn) throws SQLException {
-    /* we can cache writeIDs based on dbName and tblName because txnid is invariant and
-    partitionName is not part of the writeID select query */
-    Pair<String, String> dbAndTable = Pair.of(dbName, tblName);
-    if (writeIdCache.containsKey(dbAndTable)) {
-      return writeIdCache.get(dbAndTable);
-    } else {
-      Optional<Long> writeId = getWriteIdFromDb(txnid, dbConn, dbName, tblName);
-      writeIdCache.put(dbAndTable, writeId);
-      return writeId;
-    }
-  }
-
-  private Optional<Long> getWriteIdFromDb(long txnid, Connection dbConn, String dbName, String tblName) throws SQLException {
-    if (tblName != null) {
-      // It is assumed the caller have already allocated write id for adding/updating data to
-      // the acid tables. However, DDL operatons won't allocate write id and hence this query
-      // may return empty result sets.
-      // Get the write id allocated by this txn for the given table writes
-      try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_WRITE_ID_QUERY)) {
-        pstmt.setString(1, dbName);
-        pstmt.setString(2, tblName);
-        pstmt.setLong(3, txnid);
-        LOG.debug("Going to execute query <{}>", SELECT_WRITE_ID_QUERY);
-        try (ResultSet rs = pstmt.executeQuery()) {
-          if (rs.next()) {
-            return Optional.of(rs.getLong(1));
-          }
-        }
-      }
-    }
-    return Optional.empty();
-  }
-
-  private boolean shouldUpdateTxnComponent(long txnid, LockRequest rqst, LockComponent lc) {
-    if(!lc.isSetOperationType()) {
-      //request came from old version of the client
-      return true; //this matches old behavior
-    }
-    else {
-      switch (lc.getOperationType()) {
-        case INSERT:
-        case UPDATE:
-        case DELETE:
-          return true;
-        case SELECT:
-          return false;
-        case NO_TXN:
-              /*this constant is a bit of a misnomer since we now always have a txn context.  It
-               just means the operation is such that we don't care what tables/partitions it
-               affected as it doesn't trigger a compaction or conflict detection.  A better name
-               would be NON_TRANSACTIONAL.*/
-          return false;
-        default:
-          //since we have an open transaction, only 4 values above are expected
-          throw new IllegalStateException("Unexpected DataOperationType: " + lc.getOperationType()
-                  + " agentInfo=" + rqst.getAgentInfo() + " " + JavaUtils.txnIdToString(txnid));
-      }
-    }
-  }
-
-  private long insertHiveLocksWithTemporaryExtLockId(long txnid, Connection dbConn, LockRequest rqst) throws MetaException, SQLException {
-
-    String lastHB = isValidTxn(txnid) ? "0" : getEpochFn(dbProduct);
-    String insertLocksQuery = String.format(HIVE_LOCKS_INSERT_QRY, lastHB);
-    long intLockId = 0;
-    long tempExtLockId = generateTemporaryId();
-
-    try (PreparedStatement pstmt = dbConn.prepareStatement(insertLocksQuery)) {
-      for (LockComponent lc : rqst.getComponent()) {
-        intLockId++;
-        String lockType = LockTypeUtil.getEncodingAsStr(lc.getType());
-
-        pstmt.setLong(1, tempExtLockId);
-        pstmt.setLong(2, intLockId);
-        pstmt.setLong(3, txnid);
-        pstmt.setString(4, normalizeCase(lc.getDbname()));
-        pstmt.setString(5, normalizeCase(lc.getTablename()));
-        pstmt.setString(6, normalizePartitionCase(lc.getPartitionname()));
-        pstmt.setString(7, Character.toString(LOCK_WAITING));
-        pstmt.setString(8, lockType);
-        pstmt.setString(9, rqst.getUser());
-        pstmt.setString(10, rqst.getHostname());
-        pstmt.setString(11, rqst.getAgentInfo());
-
-        pstmt.addBatch();
-        if (intLockId % maxBatchSize == 0) {
-          LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, maxBatchSize);
-          pstmt.executeBatch();
-        }
-      }
-      if (intLockId % maxBatchSize != 0) {
-        LOG.debug("Executing a batch of <{}> queries. Batch size: {}", insertLocksQuery, intLockId % maxBatchSize);
-        pstmt.executeBatch();
-      }
-    }
-    return tempExtLockId;
-  }
-
-  private long generateTemporaryId() {
-    return -1 * ThreadLocalRandom.current().nextLong();
-  }
-
-  private static String normalizeCase(String s) {
-    return s == null ? null : s.toLowerCase();
-  }
-
-  private static String normalizePartitionCase(String s) {
-    if (s == null) {
-      return null;
-    }
-    Map<String, String> map = Splitter.on(Path.SEPARATOR).withKeyValueSeparator('=').split(s);
-    return FileUtils.makePartName(new ArrayList<>(map.keySet()), new ArrayList<>(map.values()));
-  }
-
-  private LockResponse checkLockWithRetry(Connection dbConn, long extLockId, long txnId, boolean zeroWaitReadEnabled, 
-          boolean isExclusiveCTAS)
-      throws NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      try {
-        lockInternal();
-        if(dbConn.isClosed()) {
-          //should only get here if retrying this op
-          dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        }
-        return checkLock(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for extLockId={}/txnId={}. Exception msg: {}", extLockId, txnId, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLockWithRetry(" + extLockId + "," + txnId + ")");
-        throw new MetaException("Unable to update transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        unlockInternal();
-        closeDbConn(dbConn);
-      }
-    }
-    catch(RetryException e) {
-      LOG.debug("Going to retry checkLock for extLockId={}/txnId={} after catching RetryException with message: {}",
-              extLockId, txnId, e.getMessage());
-      return checkLockWithRetry(dbConn, extLockId, txnId, zeroWaitReadEnabled, isExclusiveCTAS);
-    }
-  }
-  /**
-   * Why doesn't this get a txnid as parameter?  The caller should either know the txnid or know there isn't one.
-   * Either way getTxnIdFromLockId() will not be needed.  This would be a Thrift change.
-   *
-   * Also, when lock acquisition returns WAITING, it's retried every 15 seconds (best case, see DbLockManager.backoff(),
-   * in practice more often)
-   * which means this is heartbeating way more often than hive.txn.timeout and creating extra load on DB.
-   *
-   * The clients that operate in blocking mode, can't heartbeat a lock until the lock is acquired.
-   * We should make CheckLockRequest include timestamp or last request to skip unnecessary heartbeats. Thrift change.
-   *
-   * {@link #checkLock(java.sql.Connection, long, long, boolean, boolean)}  must run at SERIALIZABLE
-   * (make sure some lock we are checking against doesn't move from W to A in another txn)
-   * but this method can heartbeat in separate txn at READ_COMMITTED.
-   *
-   * Retry-by-caller note:
-   * Retryable because {@link #checkLock(Connection, long, long, boolean, boolean)} is
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public LockResponse checkLock(CheckLockRequest rqst)
-    throws NoSuchTxnException, NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      long extLockId = rqst.getLockid();
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        // Heartbeat on the lockid first, to assure that our lock is still valid.
-        // Then look up the lock info (hopefully in the cache).  If these locks
-        // are associated with a transaction then heartbeat on that as well.
-        LockInfo lockInfo = getLockFromLockId(dbConn, extLockId)
-                .orElseThrow(() -> new NoSuchLockException("No such lock " + JavaUtils.lockIdToString(extLockId)));
-        if (lockInfo.txnId > 0) {
-          heartbeatTxn(dbConn, lockInfo.txnId);
-        }
-        else {
-          heartbeatLock(dbConn, extLockId);
-        }
-        //todo: strictly speaking there is a bug here.  heartbeat*() commits but both heartbeat and
-        //checkLock() are in the same retry block, so if checkLock() throws, heartbeat is also retired
-        //extra heartbeat is logically harmless, but ...
-        return checkLock(dbConn, extLockId, lockInfo.txnId, false, false);
-      } catch (SQLException e) {
-        LOG.error("checkLock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "checkLock(" + rqst + " )");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      LOG.debug("Going to retry checkLock for request={} after catching RetryException with message: {}",
-              rqst, e.getMessage());
-      return checkLock(rqst);
-    }
-
-  }
-
-  /**
-   * This would have been made simpler if all locks were associated with a txn.  Then only txn needs to
-   * be heartbeated, committed, etc.  no need for client to track individual locks.
-   * When removing locks not associated with txn this potentially conflicts with
-   * heartbeat/performTimeout which are update/delete of HIVE_LOCKS thus will be locked as needed by db.
-   * since this only removes from HIVE_LOCKS at worst some lock acquire is delayed
-   */
-  @RetrySemantics.Idempotent
-  public void unlock(UnlockRequest rqst) throws TxnOpenException, MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      long extLockId = rqst.getLockid();
-      try {
-        /**
-         * This method is logically like commit for read-only auto commit queries.
-         * READ_COMMITTED since this only has 1 delete statement and no new entries with the
-         * same hl_lock_ext_id can be added, i.e. all rows with a given hl_lock_ext_id are
-         * created in a single atomic operation.
-         * Theoretically, this competes with {@link #lock(org.apache.hadoop.hive.metastore.api.LockRequest)}
-         * but hl_lock_ext_id is not known until that method returns.
-         * Also competes with {@link #checkLock(org.apache.hadoop.hive.metastore.api.CheckLockRequest)}
-         * but using SERIALIZABLE doesn't materially change the interaction.
-         * If "delete" stmt misses, additional logic is best effort to produce meaningful error msg.
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        //hl_txnid <> 0 means it's associated with a transaction
-        String s = "DELETE FROM \"HIVE_LOCKS\" WHERE \"HL_LOCK_EXT_ID\" = " + extLockId + " AND (\"HL_TXNID\" = 0 OR" +
-          " (\"HL_TXNID\" <> 0 AND \"HL_LOCK_STATE\" = '" + LOCK_WAITING + "'))";
-        //(hl_txnid <> 0 AND hl_lock_state = '" + LOCK_WAITING + "') is for multi-statement txns where
-        //some query attempted to lock (thus LOCK_WAITING state) but is giving up due to timeout for example
-        LOG.debug("Going to execute update <{}>", s);
-        int rc = stmt.executeUpdate(s);
-        if (rc < 1) {
-          LOG.info("Failure to unlock any locks with extLockId={}.", extLockId);
-          dbConn.rollback();
-          Optional<LockInfo> optLockInfo = getLockFromLockId(dbConn, extLockId);
-          if (!optLockInfo.isPresent()) {
-            //didn't find any lock with extLockId but at ReadCommitted there is a possibility that
-            //it existed when above delete ran but it didn't have the expected state.
-            LOG.info("No lock in {} mode found for unlock({})", LOCK_WAITING,
-                JavaUtils.lockIdToString(rqst.getLockid()));
-            //bail here to make the operation idempotent
-            return;
-          }
-          LockInfo lockInfo = optLockInfo.get();
-          if (isValidTxn(lockInfo.txnId)) {
-            String msg = "Unlocking locks associated with transaction not permitted.  " + lockInfo;
-            //if a lock is associated with a txn we can only "unlock" if if it's in WAITING state
-            // which really means that the caller wants to give up waiting for the lock
-            LOG.error(msg);
-            throw new TxnOpenException(msg);
-          } else {
-            //we didn't see this lock when running DELETE stmt above but now it showed up
-            //so should "should never happen" happened...
-            String msg = "Found lock in unexpected state " + lockInfo;
-            LOG.error(msg);
-            throw new MetaException(msg);
-          }
-        }
-        LOG.debug("Successfully unlocked at least 1 lock with extLockId={}", extLockId);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.error("Unlock failed for request={}. Exception msg: {}", rqst, getMessage(e));
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "unlock(" + rqst + ")");
-        throw new MetaException("Unable to update transaction database " +
-          JavaUtils.lockIdToString(extLockId) + " " + StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      unlock(rqst);
-    }
-  }
-
-  /**
-   * used to sort entries in {@link org.apache.hadoop.hive.metastore.api.ShowLocksResponse}
-   */
-  private static class LockInfoExt extends LockInfo {
-    private final ShowLocksResponseElement e;
-    LockInfoExt(ShowLocksResponseElement e) {
-      super(e);
-      this.e = e;
-    }
-  }
-  @RetrySemantics.ReadOnly
-  public ShowLocksResponse showLocks(ShowLocksRequest rqst) throws MetaException {
-    try {
-      Connection dbConn = null;
-      ShowLocksResponse rsp = new ShowLocksResponse();
-      List<ShowLocksResponseElement> elems = new ArrayList<>();
-      List<LockInfoExt> sortedList = new ArrayList<>();
-      PreparedStatement pst = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String s = "SELECT \"HL_LOCK_EXT_ID\", \"HL_TXNID\", \"HL_DB\", \"HL_TABLE\", \"HL_PARTITION\", \"HL_LOCK_STATE\", " +
-          "\"HL_LOCK_TYPE\", \"HL_LAST_HEARTBEAT\", \"HL_ACQUIRED_AT\", \"HL_USER\", \"HL_HOST\", \"HL_LOCK_INT_ID\"," +
-          "\"HL_BLOCKEDBY_EXT_ID\", \"HL_BLOCKEDBY_INT_ID\", \"HL_AGENT_INFO\" FROM \"HIVE_LOCKS\"";
-
-        // Some filters may have been specified in the SHOW LOCKS statement. Add them to the query.
-        String dbName = rqst.getDbname();
-        String tableName = rqst.getTablename();
-        String partName = rqst.getPartname();
-        List<String> params = new ArrayList<>();
-
-        StringBuilder filter = new StringBuilder();
-        if (dbName != null && !dbName.isEmpty()) {
-          filter.append("\"HL_DB\"=?");
-          params.add(dbName);
-        }
-        if (tableName != null && !tableName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TABLE\"=?");
-          params.add(tableName);
-        }
-        if (partName != null && !partName.isEmpty()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_PARTITION\"=?");
-          params.add(partName);
-        }
-        if (rqst.isSetTxnid()) {
-          if (filter.length() > 0) {
-            filter.append(" and ");
-          }
-          filter.append("\"HL_TXNID\"=" + rqst.getTxnid());
-        }
-        String whereClause = filter.toString();
-
-        if (!whereClause.isEmpty()) {
-          s = s + " where " + whereClause;
-        }
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params);
-        LOG.debug("Going to execute query <{}>", s);
-        ResultSet rs = pst.executeQuery();
-        while (rs.next()) {
-          ShowLocksResponseElement e = new ShowLocksResponseElement();
-          e.setLockid(rs.getLong(1));
-          long txnid = rs.getLong(2);
-          if (!rs.wasNull()) e.setTxnid(txnid);
-          e.setDbname(rs.getString(3));
-          e.setTablename(rs.getString(4));
-          String partition = rs.getString(5);
-          if (partition != null) e.setPartname(partition);
-          switch (rs.getString(6).charAt(0)) {
-            case LOCK_ACQUIRED: e.setState(LockState.ACQUIRED); break;
-            case LOCK_WAITING: e.setState(LockState.WAITING); break;
-            default: throw new MetaException("Unknown lock state " + rs.getString(6).charAt(0));
-          }
-
-          char lockChar = rs.getString(7).charAt(0);
-          LockType lockType = LockTypeUtil.getLockTypeFromEncoding(lockChar)
-                  .orElseThrow(() -> new MetaException("Unknown lock type: " + lockChar));
-          e.setType(lockType);
-
-          e.setLastheartbeat(rs.getLong(8));
-          long acquiredAt = rs.getLong(9);
-          if (!rs.wasNull()) e.setAcquiredat(acquiredAt);
-          e.setUser(rs.getString(10));
-          e.setHostname(rs.getString(11));
-          e.setLockIdInternal(rs.getLong(12));
-          long id = rs.getLong(13);
-          if(!rs.wasNull()) {
-            e.setBlockedByExtId(id);
-          }
-          id = rs.getLong(14);
-          if(!rs.wasNull()) {
-            e.setBlockedByIntId(id);
-          }
-          e.setAgentInfo(rs.getString(15));
-          sortedList.add(new LockInfoExt(e));
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showLocks(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeStmt(pst);
-        closeDbConn(dbConn);
-      }
-      //this ensures that "SHOW LOCKS" prints the locks in the same order as they are examined
-      //by checkLock() - makes diagnostics easier.
-      Collections.sort(sortedList, new LockInfoComparator());
-      for(LockInfoExt lockInfoExt : sortedList) {
-        elems.add(lockInfoExt.e);
-      }
-      rsp.setLocks(elems);
-      return rsp;
-    } catch (RetryException e) {
-      return showLocks(rqst);
-    }
-  }
-
-  /**
-   * {@code ids} should only have txnid or lockid but not both, ideally.
-   * Currently DBTxnManager.heartbeat() enforces this.
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void heartbeat(HeartbeatRequest ids)
-    throws NoSuchTxnException,  NoSuchLockException, TxnAbortedException, MetaException {
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        heartbeatLock(dbConn, ids.getLockid());
-        heartbeatTxn(dbConn, ids.getTxnid());
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeat(" + ids + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      heartbeat(ids);
-    }
-  }
-  @Override
-  @RetrySemantics.SafeToRetry
-  public HeartbeatTxnRangeResponse heartbeatTxnRange(HeartbeatTxnRangeRequest rqst)
-    throws MetaException {
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      HeartbeatTxnRangeResponse rsp = new HeartbeatTxnRangeResponse();
-      Set<Long> nosuch = new HashSet<>();
-      Set<Long> aborted = new HashSet<>();
-      rsp.setNosuch(nosuch);
-      rsp.setAborted(aborted);
-      try {
-        /**
-         * READ_COMMITTED is sufficient since {@link #heartbeatTxn(java.sql.Connection, long)}
-         * only has 1 update statement in it and
-         * we only update existing txns, i.e. nothing can add additional txns that this operation
-         * would care about (which would have required SERIALIZABLE)
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        /*do fast path first (in 1 statement) if doesn't work, rollback and do the long version*/
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        int numTxnsToHeartbeat = (int) (rqst.getMax() - rqst.getMin() + 1);
-        List<Long> txnIds = new ArrayList<>(numTxnsToHeartbeat);
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          txnIds.add(txn);
-        }
-        TxnUtils.buildQueryWithINClause(conf, queries,
-          new StringBuilder("UPDATE \"TXNS\" SET \"TXN_LAST_HEARTBEAT\" = " + getEpochFn(dbProduct) +
-            " WHERE \"TXN_STATE\" = " + TxnStatus.OPEN + " AND "),
-          new StringBuilder(""), txnIds, "\"TXN_ID\"", true, false);
-        int updateCnt = 0;
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          updateCnt += stmt.executeUpdate(query);
-        }
-        if (updateCnt == numTxnsToHeartbeat) {
-          //fast pass worked, i.e. all txns we were asked to heartbeat were Open as expected
-          dbConn.commit();
-          return rsp;
-        }
-        //if here, do the slow path so that we can return info txns which were not in expected state
-        dbConn.rollback();
-        for (long txn = rqst.getMin(); txn <= rqst.getMax(); txn++) {
-          try {
-            heartbeatTxn(dbConn, txn);
-          } catch (NoSuchTxnException e) {
-            nosuch.add(txn);
-          } catch (TxnAbortedException e) {
-            aborted.add(txn);
-          }
-        }
-        return rsp;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "heartbeatTxnRange(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-      }
-    } catch (RetryException e) {
-      return heartbeatTxnRange(rqst);
-    }
-  }
-
-  @Deprecated
-  long generateCompactionQueueId(Statement stmt) throws SQLException, MetaException {
-    // Get the id for the next entry in the queue
-    String s = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute query <{}>", s);
-    try (ResultSet rs = stmt.executeQuery(s)) {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      s = "UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = " + (id + 1) + " WHERE \"NCQ_NEXT\" = " + id;
-      LOG.debug("Going to execute update <{}>", s);
-      if (stmt.executeUpdate(s) != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return generateCompactionQueueId(stmt);
-      }
-      return id;
-    }
-  }
-
-  long generateCompactionQueueId() throws MetaException {
-    // Get the id for the next entry in the queue
-    String sql = sqlGenerator.addForUpdateClause("SELECT \"NCQ_NEXT\" FROM \"NEXT_COMPACTION_QUEUE_ID\"");
-    LOG.debug("going to execute SQL <{}>", sql);
-    
-    Long allocatedId = jdbcResource.getJdbcTemplate().query(sql, rs -> {
-      if (!rs.next()) {
-        throw new IllegalStateException("Transaction tables not properly initiated, "
-            + "no record found in next_compaction_queue_id");
-      }
-      long id = rs.getLong(1);
-      
-      int count = jdbcResource.getJdbcTemplate().update("UPDATE \"NEXT_COMPACTION_QUEUE_ID\" SET \"NCQ_NEXT\" = :newId WHERE \"NCQ_NEXT\" = :id",
-          new MapSqlParameterSource()
-              .addValue("id", id)
-              .addValue("newId", id + 1));
-      
-      if (count != 1) {
-        //TODO: Eliminate this id generation by implementing: https://issues.apache.org/jira/browse/HIVE-27121
-        LOG.info("The returned compaction ID ({}) already taken, obtaining new", id);
-        return null;
-      }
-      return id;
-    });
-    if (allocatedId == null) {
-      return generateCompactionQueueId();
-    } else {
-      return allocatedId;
-    }
-  }
-
-
-  @Override
-  @RetrySemantics.ReadOnly
-  public long getTxnIdForWriteId(
-      String dbName, String tblName, long writeId) throws MetaException {
-    try {
-      Connection dbConn = null;
-      PreparedStatement pst = null;
-      try {
-        /**
-         * This runs at READ_COMMITTED for exactly the same reason as {@link #getOpenTxnsInfo()}
-         */
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        String query = "SELECT \"T2W_TXNID\" FROM \"TXN_TO_WRITE_ID\" WHERE"
-            + " \"T2W_DATABASE\" = ? AND \"T2W_TABLE\" = ? AND \"T2W_WRITEID\" = " + writeId;
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, query, Arrays.asList(dbName, tblName));
-        if (LOG.isDebugEnabled()) {
-          LOG.debug("Going to execute query <" + query.replace("?", "{}") + ">",
-              quoteString(dbName), quoteString(tblName));
-        }
-        ResultSet rs  = pst.executeQuery();
-        long txnId = -1;
-        if (rs.next()) {
-          txnId = rs.getLong(1);
-        }
-        return txnId;
-      } catch (SQLException e) {
-        checkRetryable(e, "getTxnIdForWriteId");
-        throw new MetaException("Unable to select from transaction database, "
-                + StringUtils.stringifyException(e));
-      } finally {
-        close(null, pst, dbConn);
-      }
-    } catch (RetryException e) {
-      return getTxnIdForWriteId(dbName, tblName, writeId);
-    }
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public CompactionResponse compact(CompactionRequest rqst) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      TxnStore.MutexAPI.LockHandle handle = null;
-      try {
-        lockInternal();
-        /**
-         * MUTEX_KEY.CompactionScheduler lock ensures that there is only 1 entry in
-         * Initiated/Working state for any resource.  This ensures that we don't run concurrent
-         * compactions for any resource.
-         */
-        handle = getMutexAPI().acquireLock(MUTEX_KEY.CompactionScheduler.name());
-
-        try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED)) {
-          try (Statement stmt = dbConn.createStatement()) {
-
-            long id = generateCompactionQueueId(stmt);
-
-            GetValidWriteIdsRequest request = new GetValidWriteIdsRequest(
-                Collections.singletonList(getFullTableName(rqst.getDbname(), rqst.getTablename())));
-            final ValidCompactorWriteIdList tblValidWriteIds =
-                TxnUtils.createValidCompactWriteIdList(getValidWriteIds(request).getTblValidWriteIds().get(0));
-            if (LOG.isDebugEnabled()) {
-              LOG.debug("ValidCompactWriteIdList: {}", tblValidWriteIds.writeToString());
-            }
-
-            StringBuilder sb = new StringBuilder("SELECT \"CQ_ID\", \"CQ_STATE\" FROM \"COMPACTION_QUEUE\" WHERE").
-                append(" (\"CQ_STATE\" IN(").
-                append(quoteChar(INITIATED_STATE)).append(",").append(quoteChar(WORKING_STATE)).
-                append(") OR (\"CQ_STATE\" = ").append(quoteChar(READY_FOR_CLEANING)).
-                append(" AND \"CQ_HIGHEST_WRITE_ID\" = ?))").
-                append(" AND \"CQ_DATABASE\"=?").
-                append(" AND \"CQ_TABLE\"=?").append(" AND ");
-            if(rqst.getPartitionname() == null) {
-              sb.append("\"CQ_PARTITION\" is null");
-            } else {
-              sb.append("\"CQ_PARTITION\"=?");
-            }
-
-            try (PreparedStatement pst = dbConn.prepareStatement(sqlGenerator.addEscapeCharacters(sb.toString()))) {
-              pst.setLong(1, tblValidWriteIds.getHighWatermark());
-              pst.setString(2, rqst.getDbname());
-              pst.setString(3, rqst.getTablename());
-              if (rqst.getPartitionname() != null) {
-                pst.setString(4, rqst.getPartitionname());
-              }
-              LOG.debug("Going to execute query <{}>", sb);
-              try (ResultSet rs = pst.executeQuery()) {
-                if(rs.next()) {
-                  long enqueuedId = rs.getLong(1);
-                  String state = CompactionState.fromSqlConst(rs.getString(2)).toString();
-                  LOG.info("Ignoring request to compact {}/{}/{} since it is already {} with id={}", rqst.getDbname(),
-                      rqst.getTablename(), rqst.getPartitionname(), quoteString(state), enqueuedId);
-                  CompactionResponse resp = new CompactionResponse(-1, REFUSED_RESPONSE, false);
-                  resp.setErrormessage("Compaction is already scheduled with state=" + quoteString(state) +
-                      " and id=" + enqueuedId);
-                  return resp;
-                }
-              }
-            }
-            List<String> params = new ArrayList<>();
-            StringBuilder buf = new StringBuilder("INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_DATABASE\", " +
-                "\"CQ_TABLE\", ");
-            String partName = rqst.getPartitionname();
-            if (partName != null) buf.append("\"CQ_PARTITION\", ");
-            buf.append("\"CQ_STATE\", \"CQ_TYPE\", \"CQ_ENQUEUE_TIME\", \"CQ_POOL_NAME\"");
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", \"CQ_NUMBER_OF_BUCKETS\"");
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", \"CQ_ORDER_BY\"");
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", \"CQ_TBLPROPERTIES\"");
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", \"CQ_RUN_AS\"");
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", \"CQ_INITIATOR_ID\"");
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", \"CQ_INITIATOR_VERSION\"");
-            }
-            buf.append(") values (");
-            buf.append(id);
-            buf.append(", ?");
-            buf.append(", ?");
-            buf.append(", ");
-            params.add(rqst.getDbname());
-            params.add(rqst.getTablename());
-            if (partName != null) {
-              buf.append("?, '");
-              params.add(partName);
-            } else {
-              buf.append("'");
-            }
-            buf.append(INITIATED_STATE);
-            buf.append("', '");
-            buf.append(TxnUtils.thriftCompactionType2DbType(rqst.getType()));
-            buf.append("',");
-            buf.append(getEpochFn(dbProduct));
-            buf.append(", ?");
-            params.add(rqst.getPoolName());
-            if (rqst.isSetNumberOfBuckets()) {
-              buf.append(", ").append(rqst.getNumberOfBuckets());
-            }
-            if (rqst.isSetOrderByClause()) {
-              buf.append(", ?");
-              params.add(rqst.getOrderByClause());
-            }
-            if (rqst.getProperties() != null) {
-              buf.append(", ?");
-              params.add(new StringableMap(rqst.getProperties()).toString());
-            }
-            if (rqst.getRunas() != null) {
-              buf.append(", ?");
-              params.add(rqst.getRunas());
-            }
-            if (rqst.getInitiatorId() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorId());
-            }
-            if (rqst.getInitiatorVersion() != null) {
-              buf.append(", ?");
-              params.add(rqst.getInitiatorVersion());
-            }
-            buf.append(")");
-            String s = buf.toString();
-
-            try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-              LOG.debug("Going to execute update <{}>", s);
-              pst.executeUpdate();
-            }
-            LOG.debug("Going to commit");
-            dbConn.commit();
-            return new CompactionResponse(id, INITIATED_RESPONSE, true);
-          } catch (SQLException e) {
-            LOG.debug("Going to rollback: ", e);
-            dbConn.rollback();
-            throw e;
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "COMPACT(" + rqst + ")");
-        throw new MetaException("Unable to put the compaction request into the queue: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        if (handle != null) {
-          handle.releaseLocks();
-        }
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return compact(rqst);
-    }
-  }
-
-  @Override
-  @RetrySemantics.SafeToRetry
-  public boolean submitForCleanup(CompactionRequest rqst, long highestWriteId, long txnId) throws MetaException {
-    // Put a compaction request in the queue.
-    try {
-      Connection dbConn = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        lockInternal();
-
-        List<String> params = new ArrayList<String>() {{
-          add(rqst.getDbname());
-          add(rqst.getTablename());
-        }};
-        long cqId;
-        try (Statement stmt = dbConn.createStatement()) {
-          cqId = generateCompactionQueueId(stmt);
-        }
-        StringBuilder buf = new StringBuilder(
-            "INSERT INTO \"COMPACTION_QUEUE\" (\"CQ_ID\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TXN_ID\", \"CQ_ENQUEUE_TIME\", \"CQ_DATABASE\", \"CQ_TABLE\", ");
-        String partName = rqst.getPartitionname();
-        if (partName != null) {
-          buf.append("\"CQ_PARTITION\", ");
-          params.add(partName);
-        }
-        buf.append("\"CQ_STATE\", \"CQ_TYPE\"");
-        params.add(String.valueOf(READY_FOR_CLEANING));
-        params.add(TxnUtils.thriftCompactionType2DbType(rqst.getType()).toString());
-
-        if (rqst.getProperties() != null) {
-          buf.append(", \"CQ_TBLPROPERTIES\"");
-          params.add(new StringableMap(rqst.getProperties()).toString());
-        }
-        if (rqst.getRunas() != null) {
-          buf.append(", \"CQ_RUN_AS\"");
-          params.add(rqst.getRunas());
-        }
-        buf.append(") values (")
-          .append(
-            Stream.of(cqId, highestWriteId, txnId, getEpochFn(dbProduct))
-              .map(Object::toString)
-              .collect(Collectors.joining(", ")))
-          .append(repeat(", ?", params.size()))
-          .append(")");
-
-        String s = buf.toString();
-        try (PreparedStatement pst = sqlGenerator.prepareStmtWithParameters(dbConn, s, params)) {
-          LOG.debug("Going to execute update <{}>", s);
-          pst.executeUpdate();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-        return true;
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "submitForCleanup(" + rqst + ")");
-        throw new MetaException("Failed to submit cleanup request: " +
-          StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      return submitForCleanup(rqst, highestWriteId, txnId);
-    }
-  }
-  
-  @RetrySemantics.ReadOnly
-  @SuppressWarnings("squid:S2095")
-  public ShowCompactResponse showCompact(ShowCompactRequest rqst) throws MetaException {
-    try {
-      ShowCompactResponse response = new ShowCompactResponse(new ArrayList<>());
-      String query = TxnQueries.SHOW_COMPACTION_QUERY +
-        getShowCompactFilterClause(rqst) +
-        getShowCompactSortingOrderClause(rqst);
-      List<String> params = getShowCompactParamList(rqst);
-
-      try (Connection dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        PreparedStatement stmt = sqlGenerator.prepareStmtWithParameters(dbConn, query, params)) {
-        if (rqst.isSetId()) {
-          stmt.setLong(1, rqst.getId());
-        }
-        int rowLimit = (int) rqst.getLimit();
-        if (rowLimit > 0) {
-          stmt.setMaxRows(rowLimit);
-        }
-        LOG.debug("Going to execute query <" + query + ">");
-        try (ResultSet rs = stmt.executeQuery()) {
-          while (rs.next()) {
-            ShowCompactResponseElement e = new ShowCompactResponseElement();
-            e.setDbname(rs.getString(1));
-            e.setTablename(rs.getString(2));
-            e.setPartitionname(rs.getString(3));
-            e.setState(CompactionState.fromSqlConst(rs.getString(4)).toString());
-            try {
-              e.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-            } catch (SQLException ex) {
-              //do nothing to handle RU/D if we add another status
-            }
-            e.setWorkerid(rs.getString(6));
-            long start = rs.getLong(7);
-            if (!rs.wasNull()) {
-              e.setStart(start);
-            }
-            long endTime = rs.getLong(8);
-            if (endTime != -1) {
-              e.setEndTime(endTime);
-            }
-            e.setRunAs(rs.getString(9));
-            e.setHadoopJobId(rs.getString(10));
-            e.setId(rs.getLong(11));
-            e.setErrorMessage(rs.getString(12));
-            long enqueueTime = rs.getLong(13);
-            if (!rs.wasNull()) {
-              e.setEnqueueTime(enqueueTime);
-            }
-            e.setWorkerVersion(rs.getString(14));
-            e.setInitiatorId(rs.getString(15));
-            e.setInitiatorVersion(rs.getString(16));
-            long cleanerStart = rs.getLong(17);
-            if (!rs.wasNull() && (cleanerStart != -1)) {
-              e.setCleanerStart(cleanerStart);
-            }
-            String poolName = rs.getString(18);
-            if (isBlank(poolName)) {
-              e.setPoolName(DEFAULT_POOL_NAME);
-            } else {
-              e.setPoolName(poolName);
-            }
-            e.setTxnId(rs.getLong(19));
-            e.setNextTxnId(rs.getLong(20));
-            e.setCommitTime(rs.getLong(21));
-            e.setHightestTxnId(rs.getLong(22));
-            response.addToCompacts(e);
-          }
-        }
-      } catch (SQLException e) {
-        checkRetryable(e, "showCompact(" + rqst + ")");
-        throw new MetaException("Unable to select from transaction database " +
-            StringUtils.stringifyException(e));
-      }
-      return response;
-    } catch (RetryException e) {
-      return showCompact(rqst);
-    }
-  }
-
-  private String getShowCompactSortingOrderClause(ShowCompactRequest request) {
-    String sortingOrder = request.getOrder();
-    return isNotBlank(sortingOrder) ? "  ORDER BY  " + sortingOrder : TxnQueries.SHOW_COMPACTION_ORDERBY_CLAUSE;
-  }
-
-  private List<String> getShowCompactParamList(ShowCompactRequest request) throws MetaException {
-    if (request.getId() > 0) {
-      return Collections.emptyList();
-    }
-    String poolName = request.getPoolName();
-    String dbName = request.getDbName();
-    String tableName = request.getTbName();
-    String partName = request.getPartName();
-    CompactionType type = request.getType();
-    String state = request.getState();
-  
-    List<String> params = new ArrayList<>();
-    if (isNotBlank(dbName)) {
-      params.add(dbName);
-    }
-    if (isNotBlank(tableName)) {
-      params.add(tableName);
-    }
-    if (isNotBlank(partName)) {
-      params.add(partName);
-    }
-    if (isNotBlank(state)) {
-      params.add(state);
-    }
-    if (type != null) {
-      params.add(TxnUtils.thriftCompactionType2DbType(type).toString());
-    }
-    if (isNotBlank(poolName)) {
-      params.add(poolName);
-    }
-    return params;
-  }
-
-  private String getShowCompactFilterClause(ShowCompactRequest request) {
-    List<String> params = new ArrayList<>();
-    
-    if (request.getId() > 0) {
-      params.add("\"CC_ID\"=?");
-    } else {
-      if (isNotBlank(request.getDbName())) {
-        params.add("\"CC_DATABASE\"=?");
-      }
-      if (isNotBlank(request.getTbName())) {
-        params.add("\"CC_TABLE\"=?");
-      }
-      if (isNotBlank(request.getPartName())) {
-        params.add("\"CC_PARTITION\"=?");
-      }
-      if (isNotBlank(request.getState())) {
-        params.add("\"CC_STATE\"=?");
-      }
-      if (request.getType() != null) {
-        params.add("\"CC_TYPE\"=?");
-      }
-      if (isNotBlank(request.getPoolName())) {
-        params.add("\"CC_POOL_NAME\"=?");
-      }
-    }
-    return !params.isEmpty() ? 
-      " WHERE " + StringUtils.join(" AND ", params) : EMPTY;
-  }
-
-  /**
-   * We assume this is only called by metadata cache server to know if there are new base/delta files should be read.
-   * The query filters compactions by state and only returns SUCCEEDED or READY_FOR_CLEANING compactions because
-   * only these two states means there are new files ready to be read.
-   */
-  @RetrySemantics.ReadOnly
-  public GetLatestCommittedCompactionInfoResponse getLatestCommittedCompactionInfo(
-      GetLatestCommittedCompactionInfoRequest rqst) throws MetaException {
-    GetLatestCommittedCompactionInfoResponse response = new GetLatestCommittedCompactionInfoResponse(new ArrayList<>());
-    Connection dbConn = null;
-    PreparedStatement pst = null;
-    ResultSet rs = null;
-    try {
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-
-        List<String> params = new ArrayList<>();
-        // This query combines the result sets of SUCCEEDED compactions and READY_FOR_CLEANING compactions
-        // We also sort the result by CC_ID in descending order so that we can keep only the latest record
-        // according to the order in result set
-        StringBuilder sb = new StringBuilder()
-            .append("SELECT * FROM (")
-            .append("   SELECT")
-            .append("   \"CC_ID\", \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_TYPE\"")
-            .append("   FROM \"COMPLETED_COMPACTIONS\"")
-            .append("     WHERE \"CC_STATE\" = " + quoteChar(SUCCEEDED_STATE))
-            .append("   UNION ALL")
-            .append("   SELECT")
-            .append("   \"CQ_ID\" AS \"CC_ID\", \"CQ_DATABASE\" AS \"CC_DATABASE\"")
-            .append("   ,\"CQ_TABLE\" AS \"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\"")
-            .append("   ,\"CQ_TYPE\" AS \"CC_TYPE\"")
-            .append("   FROM \"COMPACTION_QUEUE\"")
-            .append("     WHERE \"CQ_STATE\" = " + quoteChar(READY_FOR_CLEANING))
-            .append(") AS compactions ")
-            .append(" WHERE \"CC_DATABASE\" = ? AND \"CC_TABLE\" = ?");
-        params.add(rqst.getDbname());
-        params.add(rqst.getTablename());
-        if (rqst.getPartitionnamesSize() > 0) {
-          sb.append(" AND \"CC_PARTITION\" IN (");
-          sb.append(String.join(",",
-              Collections.nCopies(rqst.getPartitionnamesSize(), "?")));
-          sb.append(")");
-          params.addAll(rqst.getPartitionnames());
-        }
-        if (rqst.isSetLastCompactionId()) {
-          sb.append(" AND \"CC_ID\" > ?");
-        }
-        sb.append(" ORDER BY \"CC_ID\" DESC");
-
-        pst = sqlGenerator.prepareStmtWithParameters(dbConn, sb.toString(), params);
-        if (rqst.isSetLastCompactionId()) {
-          pst.setLong(params.size() + 1, rqst.getLastCompactionId());
-        }
-        LOG.debug("Going to execute query <{}>", sb);
-        rs = pst.executeQuery();
-        Set<String> partitionSet = new HashSet<>();
-        while (rs.next()) {
-          CompactionInfoStruct lci = new CompactionInfoStruct();
-          lci.setId(rs.getLong(1));
-          lci.setDbname(rs.getString(2));
-          lci.setTablename(rs.getString(3));
-          String partition = rs.getString(4);
-          if (!rs.wasNull()) {
-            lci.setPartitionname(partition);
-          }
-          lci.setType(TxnUtils.dbCompactionType2ThriftType(rs.getString(5).charAt(0)));
-          // Only put the latest record of each partition into response
-          if (!partitionSet.contains(partition)) {
-            response.addToCompactions(lci);
-            partitionSet.add(partition);
-          }
-        }
-      } catch (SQLException e) {
-        LOG.error("Unable to execute query", e);
-        checkRetryable(e, "getLatestCommittedCompactionInfo");
-      } finally {
-        close(rs, pst, dbConn);
-      }
-      return response;
-    } catch (RetryException e) {
-      return getLatestCommittedCompactionInfo(rqst);
-    }
-  }
-
-  public MetricsInfo getMetricsInfo() throws MetaException {
-    Connection dbConn = null;
-    try {
-      MetricsInfo metrics = new MetricsInfo();
-      String s = MessageFormat.format(SELECT_METRICS_INFO_QUERY, getEpochFn(dbProduct));
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        try (Statement stmt = dbConn.createStatement()){
-          ResultSet rs = stmt.executeQuery(s);
-          if (rs.next()) {
-            metrics.setTxnToWriteIdCount(rs.getInt(1));
-            metrics.setCompletedTxnsCount(rs.getInt(2));
-            metrics.setOpenReplTxnsCount(rs.getInt(3));
-            metrics.setOldestOpenReplTxnId(rs.getInt(4));
-            metrics.setOldestOpenReplTxnAge(rs.getInt(5));
-            metrics.setOpenNonReplTxnsCount(rs.getInt(6));
-            metrics.setOldestOpenNonReplTxnId(rs.getInt(7));
-            metrics.setOldestOpenNonReplTxnAge(rs.getInt(8));
-            metrics.setAbortedTxnsCount(rs.getInt(9));
-            metrics.setOldestAbortedTxnId(rs.getInt(10));
-            metrics.setOldestAbortedTxnAge(rs.getInt(11));
-            metrics.setLocksCount(rs.getInt(12));
-            metrics.setOldestLockAge(rs.getInt(13));
-            metrics.setOldestReadyForCleaningAge(rs.getInt(14));
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(SELECT_TABLES_WITH_X_ABORTED_TXNS)) {
-          Set<String> resourceNames = new TreeSet<>();
-          pstmt.setInt(1, MetastoreConf.getIntVar(conf, ConfVars.METASTORE_ACIDMETRICS_TABLES_WITH_ABORTED_TXNS_THRESHOLD));
-          ResultSet rs = pstmt.executeQuery();
-          while (rs.next()) {
-            String resourceName = rs.getString(1) + "." + rs.getString(2);
-            String partName = rs.getString(3);
-            resourceName = partName != null ? resourceName + "#" + partName : resourceName;
-            resourceNames.add(resourceName);
-          }
-          metrics.setTablesWithXAbortedTxnsCount(resourceNames.size());
-          metrics.setTablesWithXAbortedTxns(resourceNames);
-        }
-        return metrics;
-      } catch (SQLException e) {
-        LOG.error("Unable to getMetricsInfo", e);
-        checkRetryable(e, "getMetricsInfo");
-        throw new MetaException("Unable to execute getMetricsInfo() " + StringUtils.stringifyException(e));
-      } finally {
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      return getMetricsInfo();
-    }
-  }
-
-
-  private static void shouldNeverHappen(long txnid) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid));
-  }
-  private static void shouldNeverHappen(long txnid, long extLockId, long intLockId) {
-    throw new RuntimeException("This should never happen: " + JavaUtils.txnIdToString(txnid) + " "
-      + JavaUtils.lockIdToString(extLockId) + " " + intLockId);
-  }
-
-  /**
-   * Retry-by-caller note:
-   * This may be retried after dbConn.commit.  At worst, it will create duplicate entries in
-   * TXN_COMPONENTS which won't affect anything.  See more comments in {@link #commitTxn(CommitTxnRequest)}
-   */
-  @Override
-  @RetrySemantics.SafeToRetry
-  public void addDynamicPartitions(AddDynamicPartitions rqst)
-      throws NoSuchTxnException,  TxnAbortedException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    try {
-      try {
-        lockInternal();
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        TxnType txnType = getOpenTxnTypeAndLock(stmt, rqst.getTxnid());
-        if (txnType == null) {
-          //ensures txn is still there and in expected state
-          ensureValidTxn(dbConn, rqst.getTxnid(), stmt);
-          shouldNeverHappen(rqst.getTxnid());
-        }
-        //for RU this may be null so we should default it to 'u' which is most restrictive
-        OperationType ot = OperationType.UPDATE;
-        if(rqst.isSetOperationType()) {
-          ot = OperationType.fromDataOperationType(rqst.getOperationType());
-        }
-
-        Long writeId = rqst.getWriteid();
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_INSERT_QUERY)) {
-          int insertCounter = 0;
-          for (String partName : rqst.getPartitionnames()) {
-            pstmt.setLong(1, rqst.getTxnid());
-            pstmt.setString(2, normalizeCase(rqst.getDbname()));
-            pstmt.setString(3, normalizeCase(rqst.getTablename()));
-            pstmt.setString(4, partName);
-            pstmt.setString(5, ot.getSqlConst());
-            pstmt.setObject(6, writeId);
-
-            pstmt.addBatch();
-            insertCounter++;
-            if (insertCounter % maxBatchSize == 0) {
-              LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY, maxBatchSize);
-              pstmt.executeBatch();
-            }
-          }
-          if (insertCounter % maxBatchSize != 0) {
-            LOG.debug("Executing a batch of <{}> queries. Batch size: {}", TXN_COMPONENTS_INSERT_QUERY,
-                insertCounter % maxBatchSize);
-            pstmt.executeBatch();
-          }
-        }
-        try (PreparedStatement pstmt = dbConn.prepareStatement(TXN_COMPONENTS_DP_DELETE_QUERY)) {
-          pstmt.setLong(1, rqst.getTxnid());
-          pstmt.setString(2, normalizeCase(rqst.getDbname()));
-          pstmt.setString(3, normalizeCase(rqst.getTablename()));
-          pstmt.execute();
-        }
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "addDynamicPartitions(" + rqst + ")");
-        throw new MetaException("Unable to insert into from transaction database " +
-          StringUtils.stringifyException(e));
-      } finally {
-        close(null, stmt, dbConn);
-        unlockInternal();
-      }
-    } catch (RetryException e) {
-      addDynamicPartitions(rqst);
-    }
-  }
-
-  /**
-   * Clean up corresponding records in metastore tables when corresponding object is dropped,
-   * specifically: TXN_COMPONENTS, COMPLETED_TXN_COMPONENTS, COMPACTION_QUEUE, COMPLETED_COMPACTIONS
-   * Retry-by-caller note: this is only idempotent assuming it's only called by dropTable/Db/etc
-   * operations.
-   *
-   * HIVE_LOCKS and WS_SET are cleaned up by {@link AcidHouseKeeperService}, if turned on
-   */
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData) throws MetaException {
-    cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData, 0);
-  }
-
-  @Override
-  @RetrySemantics.Idempotent
-  public void cleanupRecords(HiveObjectType type, Database db, Table table,
-        Iterator<Partition> partitionIterator, long txnId) throws MetaException {
-    cleanupRecords(type, db , table, partitionIterator, false, txnId);
-  }
-  
-  private void cleanupRecords(HiveObjectType type, Database db, Table table,
-      Iterator<Partition> partitionIterator, boolean keepTxnToWriteIdMetaData, long txnId) throws MetaException {
-
-    // cleanup should be done only for objects belonging to default catalog
-    final String defaultCatalog = getDefaultCatalog(conf);
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-
-      try {
-        String dbName;
-        String tblName;
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-        StringBuilder buff = new StringBuilder();
-
-        switch (type) {
-          case DATABASE: {
-            dbName = db.getName();
-            if(!defaultCatalog.equals(db.getCatalogName())) {
-              LOG.debug("Skipping cleanup because db: " + dbName + " belongs to catalog "
-                  + "other than default catalog: " + db.getCatalogName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-            
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TXN_ID\"!=").append(txnId);
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-            buff.append(dbName.toLowerCase());
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case TABLE: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because table: {} belongs to catalog other than default catalog: {}", tblName,
-                  table.getCatName());
-              return;
-            }
-
-            buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"TC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CTC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CQ_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            if (!keepTxnToWriteIdMetaData) {
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_TO_WRITE_ID\" WHERE \"T2W_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"T2W_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"NEXT_WRITE_ID\" WHERE \"NWI_DATABASE\"='");
-              buff.append(dbName.toLowerCase());
-              buff.append("' AND \"NWI_TABLE\"='");
-              buff.append(tblName.toLowerCase());
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-            buff.setLength(0);
-            buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-            buff.append(dbName);
-            buff.append("' AND \"CMC_TABLE\"='");
-            buff.append(tblName);
-            buff.append("'");
-            queries.add(buff.toString());
-
-            break;
-          }
-          case PARTITION: {
-            dbName = table.getDbName();
-            tblName = table.getTableName();
-            if(!defaultCatalog.equals(table.getCatName())) {
-              LOG.debug("Skipping cleanup because partitions belong to catalog other than default catalog: {}",
-                  table.getCatName());
-              return;
-            }
-
-            List<FieldSchema> partCols = table.getPartitionKeys();  // partition columns
-            List<String> partVals;                                  // partition values
-            String partName;
-
-            while (partitionIterator.hasNext()) {
-              Partition p = partitionIterator.next();
-              partVals = p.getValues();
-              partName = Warehouse.makePartName(partCols, partVals);
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"TXN_COMPONENTS\" WHERE \"TC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"TC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"TC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE \"CTC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CTC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CTC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CQ_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CQ_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPLETED_COMPACTIONS\" WHERE \"CC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-
-              buff.setLength(0);
-              buff.append("DELETE FROM \"COMPACTION_METRICS_CACHE\" WHERE \"CMC_DATABASE\"='");
-              buff.append(dbName);
-              buff.append("' AND \"CMC_TABLE\"='");
-              buff.append(tblName);
-              buff.append("' AND \"CMC_PARTITION\"='");
-              buff.append(partName);
-              buff.append("'");
-              queries.add(buff.toString());
-            }
-
-            break;
-          }
-          default: {
-            throw new MetaException("Invalid object type for cleanup: " + type);
-          }
-        }
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit");
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: ", e);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, "cleanupRecords");
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform cleanup since metastore table does not exist");
-        } else {
-          throw new MetaException("Unable to clean up " + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      cleanupRecords(type, db, table, partitionIterator, keepTxnToWriteIdMetaData);
-    }
-  }
-  /**
-   * Catalog hasn't been added to transactional tables yet, so it's passed in but not used.
-   */
-  @Override
-  public void onRename(String oldCatName, String oldDbName, String oldTabName, String oldPartName,
-      String newCatName, String newDbName, String newTabName, String newPartName)
-      throws MetaException {
-    String callSig = "onRename(" +
-        oldCatName + "," + oldDbName + "," + oldTabName + "," + oldPartName + "," +
-        newCatName + "," + newDbName + "," + newTabName + "," + newPartName + ")";
-
-    if(newPartName != null) {
-      assert oldPartName != null && oldTabName != null && oldDbName != null && oldCatName != null :
-      callSig;
-    }
-    if(newTabName != null) {
-      assert oldTabName != null && oldDbName != null && oldCatName != null : callSig;
-    }
-    if(newDbName != null) {
-      assert oldDbName != null && oldCatName != null : callSig;
-    }
-
-    try {
-      Connection dbConn = null;
-      Statement stmt = null;
-      try {
-        dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-        stmt = dbConn.createStatement();
-        List<String> queries = new ArrayList<>();
-
-        String update = "UPDATE \"TXN_COMPONENTS\" SET ";
-        String where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"TC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"TC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"TC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"TC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"TC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"TC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_TXN_COMPONENTS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CTC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CTC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CTC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CTC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CTC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"HIVE_LOCKS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"HL_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"HL_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"HL_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"HL_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"HL_DB\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"HL_DB\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_QUEUE\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CQ_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CQ_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CQ_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CQ_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CQ_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPLETED_COMPACTIONS\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"CC_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"CC_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"CC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"CC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"WRITE_SET\" SET ";
-        where = " WHERE ";
-        if(oldPartName != null) {
-          update += "\"WS_PARTITION\" = " + quoteString(newPartName) + ", ";
-          where += "\"WS_PARTITION\" = " + quoteString(oldPartName) + " AND ";
-        }
-        if(oldTabName != null) {
-          update += "\"WS_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"WS_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"WS_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"WS_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"TXN_TO_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"T2W_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"T2W_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"T2W_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"NEXT_WRITE_ID\" SET ";
-        where = " WHERE ";
-        if(oldTabName != null) {
-          update += "\"NWI_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"NWI_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if(oldDbName != null) {
-          update += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"NWI_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        update = "UPDATE \"COMPACTION_METRICS_CACHE\" SET";
-        where = " WHERE ";
-        if (oldPartName != null) {
-          update += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(newPartName)) + ", ";
-          where += "\"CMC_PARTITION\" = " + quoteString(normalizeCase(oldPartName)) + " AND ";
-        }
-        if (oldTabName != null) {
-          update += "\"CMC_TABLE\" = " + quoteString(normalizeCase(newTabName)) + ", ";
-          where += "\"CMC_TABLE\" = " + quoteString(normalizeCase(oldTabName)) + " AND ";
-        }
-        if (oldDbName != null) {
-          update += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(newDbName));
-          where += "\"CMC_DATABASE\" = " + quoteString(normalizeCase(oldDbName));
-        }
-        queries.add(update + where);
-
-        for (String query : queries) {
-          LOG.debug("Going to execute update <{}>", query);
-          stmt.executeUpdate(query);
-        }
-
-        LOG.debug("Going to commit: {}", callSig);
-        dbConn.commit();
-      } catch (SQLException e) {
-        LOG.debug("Going to rollback: {}", callSig);
-        rollbackDBConn(dbConn);
-        checkRetryable(e, callSig);
-        if (e.getMessage().contains("does not exist")) {
-          LOG.warn("Cannot perform {} since metastore table does not exist", callSig);
-        } else {
-          throw new MetaException("Unable to " + callSig + ":" + StringUtils.stringifyException(e));
-        }
-      } finally {
-        closeStmt(stmt);
-        closeDbConn(dbConn);
-      }
-    } catch (RetryException e) {
-      onRename(oldCatName, oldDbName, oldTabName, oldPartName,
-          newCatName, newDbName, newTabName, newPartName);
-    }
-  }
-  /**
-   * For testing only, do not use.
-   */
-  @VisibleForTesting
-  public int numLocksInLockTable() throws SQLException, MetaException {
-    Connection dbConn = null;
-    Statement stmt = null;
-    ResultSet rs = null;
-    try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      stmt = dbConn.createStatement();
-      String s = "SELECT COUNT(*) FROM \"HIVE_LOCKS\"";
-      LOG.debug("Going to execute query <{}>", s);
-      rs = stmt.executeQuery(s);
-      rs.next();
-      int rc = rs.getInt(1);
-      // Necessary to clean up the transaction in the db.
-      dbConn.rollback();
-      return rc;
-    } finally {
-      close(rs, stmt, dbConn);
-    }
-  }
-
-  /**
-   * For testing only, do not use.
-   */
-  public long setTimeout(long milliseconds) {
-    long previous_timeout = timeout;
-    timeout = milliseconds;
-    return previous_timeout;
-  }
-
-  protected class RetryException extends Exception {
-
-  }
-
-  Connection getDbConn(int isolationLevel) throws SQLException {
-    return getDbConn(isolationLevel, connPool);
-  }
-
-  protected Connection getDbConn(int isolationLevel, DataSource connPool) throws SQLException {
-    Connection dbConn = null;
-    try {
-      dbConn = connPool.getConnection();
-      dbConn.setAutoCommit(false);
-      dbConn.setTransactionIsolation(isolationLevel);
-      return dbConn;
-    } catch (SQLException e) {
-      closeDbConn(dbConn);
-      throw e;
-    }
-  }
-
-  static void rollbackDBConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
-    } catch (SQLException e) {
-      LOG.warn("Failed to rollback db connection " + getMessage(e));
-    }
-  }
-  protected static void closeDbConn(Connection dbConn) {
-    try {
-      if (dbConn != null && !dbConn.isClosed()) {
-        dbConn.close();
-      }
-    } catch (SQLException e) {
-      LOG.warn("Failed to close db connection " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close statement instance.
-   * @param stmt statement instance.
-   */
-  protected static void closeStmt(Statement stmt) {
-    try {
-      if (stmt != null && !stmt.isClosed()) stmt.close();
-    } catch (SQLException e) {
-      LOG.warn("Failed to close statement " + getMessage(e));
-    }
-  }
-
-  /**
-   * Close the ResultSet.
-   * @param rs may be {@code null}
-   */
-  static void close(ResultSet rs) {
-    try {
-      if (rs != null && !rs.isClosed()) {
-        rs.close();
-      }
-    }
-    catch(SQLException ex) {
-      LOG.warn("Failed to close statement " + getMessage(ex));
-    }
-  }
-
-  /**
-   * Close all 3 JDBC artifacts in order: {@code rs stmt dbConn}
-   */
-  static void close(ResultSet rs, Statement stmt, Connection dbConn) {
-    close(rs);
-    closeStmt(stmt);
-    closeDbConn(dbConn);
-  }
-
-  private boolean waitForRetry(String caller, String errMsg) {
-    if (retryNum++ < retryLimit) {
-      LOG.warn("Retryable error detected in {}. Will wait {} ms and retry up to {} times. Error: {}", caller,
-          retryInterval, (retryLimit - retryNum + 1), errMsg);
-      try {
-        Thread.sleep(retryInterval);
-      } catch (InterruptedException ex) {
-        //
-      }
-      return true;
-    } else {
-      LOG.error("Fatal error in {}. Retry limit ({}) reached. Last error: {}", caller, retryLimit, errMsg);
-    }
-    return false;
-  }
-
-  /**
-   * See {@link #checkRetryable(SQLException, String, boolean)}.
-   */
-  void checkRetryable(SQLException e, String caller) throws RetryException {
-    checkRetryable(e, caller, false);
-  }
-
-  /**
-   * Determine if an exception was such that it makes sense to retry.  Unfortunately there is no standard way to do
-   * this, so we have to inspect the error messages and catch the telltale signs for each
-   * different database.  This method will throw {@code RetryException}
-   * if the error is retry-able.
-   * @param e exception that was thrown.
-   * @param caller name of the method calling this (and other info useful to log)
-   * @param retryOnDuplicateKey whether to retry on unique key constraint violation
-   * @throws org.apache.hadoop.hive.metastore.txn.TxnHandler.RetryException when the operation should be retried
+   * there will be a duplicate set of locks but both sets will belong to the same txn so they
+   * will not conflict with each other.  For locks w/o txn context (i.e. read-only query), this
+   * may lead to deadlock (at least a long wait).  (e.g. 1st call creates locks in {@code LOCK_WAITING}
+   * mode and response gets lost.  Then {@link org.apache.hadoop.hive.metastore.RetryingMetaStoreClient}
+   * retries, and enqueues another set of locks in LOCK_WAITING.  The 2nd LockResponse is delivered
+   * to the DbLockManager, which will keep dong {@link #checkLock(CheckLockRequest)} until the 1st
+   * set of locks times out.
    */
-  void checkRetryable(SQLException e, String caller, boolean retryOnDuplicateKey)
-      throws RetryException {
-
-    // If you change this function, remove the @Ignore from TestTxnHandler.deadlockIsDetected()
-    // to test these changes.
-    // MySQL and MSSQL use 40001 as the state code for rollback.  Postgres uses 40001 and 40P01.
-    // Oracle seems to return different SQLStates and messages each time,
-    // so I've tried to capture the different error messages (there appear to be fewer different
-    // error messages than SQL states).
-    // Derby and newer MySQL driver use the new SQLTransactionRollbackException
-    boolean sendRetrySignal = false;
+  @RetrySemantics.CannotRetry
+  public LockResponse lock(LockRequest rqst) throws NoSuchTxnException, TxnAbortedException, MetaException {
     try {
-      if(dbProduct == null) {
-        throw new IllegalStateException("DB Type not determined yet.");
-      }
-      if (dbProduct.isDeadlock(e)) {
-        if (deadlockCnt++ < ALLOWED_REPEATED_DEADLOCKS) {
-          long waitInterval = deadlockRetryInterval * deadlockCnt;
-          LOG.warn("Deadlock detected in {}. Will wait {} ms try again up to {} times.", caller, waitInterval,
-              (ALLOWED_REPEATED_DEADLOCKS - deadlockCnt + 1));
-          // Pause for a just a bit for retrying to avoid immediately jumping back into the deadlock.
-          try {
-            Thread.sleep(waitInterval);
-          } catch (InterruptedException ie) {
-            // NOP
-          }
-          sendRetrySignal = true;
-        } else {
-          LOG.error("Too many repeated deadlocks in {}, giving up.", caller);
-        }
-      } else if (isRetryable(conf, e)) {
-        //in MSSQL this means Communication Link Failure
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      } else if (retryOnDuplicateKey && isDuplicateKeyError(e)) {
-        sendRetrySignal = waitForRetry(caller, e.getMessage());
-      }
-      else {
-        //make sure we know we saw an error that we don't recognize
-        LOG.info("Non-retryable error in {} : {}", caller, getMessage(e));
-      }
-    }
-    finally {
-      /*if this method ends with anything except a retry signal, the caller should fail the operation
-      and propagate the error up to the its caller (Metastore client); thus must reset retry counters*/
-      if(!sendRetrySignal) {
-        deadlockCnt = 0;
-        retryNum = 0;
-      }
-    }
-    if(sendRetrySignal) {
-      throw new RetryException();
+      return txnLockManager.checkLock(txnLockManager.enqueueLock(rqst), rqst.getTxnid(), rqst.isZeroWaitReadEnabled(),

Review Comment:
   why don't we create `lock(rqst)` method in txnLockManager? and make `enqueueLock` private



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419092662


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -1696,41 +639,7 @@ private boolean isUpdateOrDelete(Statement stmt, String conflictSQLSuffix) throw
    */
   @RetrySemantics.ReadOnly

Review Comment:
   just @Override



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418887194


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/HiveMutex.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl;
+
+import org.apache.commons.lang3.NotImplementedException;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnStore;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionContext;
+import org.apache.hadoop.hive.metastore.txn.retryhandling.SqlRetryHandler;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.apache.hadoop.util.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.dao.DataAccessException;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import java.sql.SQLException;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Semaphore;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnStore.POOL_MUTEX;
+import static org.springframework.transaction.TransactionDefinition.PROPAGATION_REQUIRED;
+
+public class HiveMutex implements TxnStore.MutexAPI {

Review Comment:
   should it be TxnStoreMutex? HiveMutex is too generic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418958428


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -470,28 +320,29 @@ private boolean checkIfTableIsUsable(String tableName, boolean configValue) thro
       // don't check it if disabled
       return false;
     }
-    Connection dbConn = null;
-    boolean tableExists = true;
+    jdbcResource.bindDataSource(POOL_TX);
     try {
-      dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
-      try (Statement stmt = dbConn.createStatement()) {
-        // Dummy query to see if table exists
-        try (ResultSet rs = stmt.executeQuery("SELECT 1 FROM \"" + tableName + "\"")) {
-          rs.next();
-        }
-      }
-    } catch (SQLException e) {
+      jdbcResource.getJdbcTemplate().query("SELECT 1 FROM \"" + tableName + "\"", new MapSqlParameterSource(),
+          rs -> {
+            if (rs.next()) {
+              return rs.getInt(1);
+            }
+            return -1;

Review Comment:
   -1 would be cast to false?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419070543


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##########
@@ -6052,450 +1096,53 @@ protected synchronized static DataSource setupJdbcConnectionPool(Configuration c
         throw new RuntimeException(e);
       }
     } else {
-      String connectionPooler = MetastoreConf.getVar(conf, ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
+      String connectionPooler = MetastoreConf.getVar(conf, MetastoreConf.ConfVars.CONNECTION_POOLING_TYPE).toLowerCase();
       if ("none".equals(connectionPooler)) {
         LOG.info("Choosing not to pool JDBC connections");
-        return new NoPoolConnectionPool(conf);
+        return new NoPoolConnectionPool(conf, dbProduct);
       } else {
         throw new RuntimeException("Unknown JDBC connection pooling " + connectionPooler);
       }
     }
   }
 
-  /**
-   * Returns true if {@code ex} should be retried
-   */
-  static boolean isRetryable(Configuration conf, Exception ex) {
-    if(ex instanceof SQLException) {
-      SQLException sqlException = (SQLException)ex;
-      if (MANUAL_RETRY.equalsIgnoreCase(sqlException.getSQLState())) {
-        // Manual retry exception was thrown
-        return true;
-      }
-      if ("08S01".equalsIgnoreCase(sqlException.getSQLState())) {
-        //in MSSQL this means Communication Link Failure
-        return true;
-      }
-      if ("ORA-08176".equalsIgnoreCase(sqlException.getSQLState()) ||
-        sqlException.getMessage().contains("consistent read failure; rollback data not available")) {
-        return true;
-      }
-
-      String regex = MetastoreConf.getVar(conf, ConfVars.TXN_RETRYABLE_SQLEX_REGEX);
-      if (regex != null && !regex.isEmpty()) {
-        String[] patterns = regex.split(",(?=\\S)");
-        String message = getMessage((SQLException)ex);
-        for (String p : patterns) {
-          if (Pattern.matches(p, message)) {
-            return true;
-          }
-        }
-      }
-      //see also https://issues.apache.org/jira/browse/HIVE-9938
-    }
-    return false;
-  }
-
-  private boolean isDuplicateKeyError(SQLException ex) {
-    return dbProduct.isDuplicateKeyError(ex);
-  }
-
-  private static String getMessage(SQLException ex) {
-    return ex.getMessage() + " (SQLState=" + ex.getSQLState() + ", ErrorCode=" + ex.getErrorCode() + ")";
-  }
-  static String quoteString(String input) {
-    return "'" + input + "'";
-  }
-  static String quoteChar(char c) {
-    return "'" + c + "'";
-  }
-
-  /**
-   * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
-   * Select ... For Update to sequence operations properly.  In practice that means when running
-   * with Derby database.  See more notes at class level.
-   */
-  protected void lockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.lock();
-    }
-  }
-  protected void unlockInternal() {
-    if(dbProduct.isDERBY()) {
-      derbyLock.unlock();
-    }
-  }
   @Override
   @RetrySemantics.Idempotent
   public MutexAPI getMutexAPI() {

Review Comment:
   should we define retry semantic here, as it's also present in interface?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418823231


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/StackThreadLocal.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn;
+
+import java.util.Stack;
+
+public class StackThreadLocal<T> {

Review Comment:
   it looks like util class, why it's under txn package?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418867491


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java:
##########
@@ -666,4 +627,60 @@ public static Character thriftCompactionType2DbType(CompactionType ct) throws Me
   public static String nvl(String input) {
     return input != null ? " = ? " : " IS NULL ";
   }
+
+  public static String normalizePartitionCase(String s) {

Review Comment:
   should we move this under `PartitionUtils`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1419174351


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/impl/functions/HeartBeatLockFunction.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.impl.functions;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchLockException;
+import org.apache.hadoop.hive.metastore.api.NoSuchTxnException;
+import org.apache.hadoop.hive.metastore.api.TxnAbortedException;
+import org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource;
+import org.apache.hadoop.hive.metastore.txn.jdbc.TransactionalFunction;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.springframework.jdbc.core.namedparam.MapSqlParameterSource;
+
+import static org.apache.hadoop.hive.metastore.txn.TxnUtils.getEpochFn;
+
+/**
+ * Heartbeats on the lock table.  This commits, so do not enter it with any state.
+ * Should not be called on a lock that belongs to transaction.
+ */
+public class HeartBeatLockFunction implements TransactionalFunction<Void> {

Review Comment:
   it's still with HeartBeat



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418953054


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -546,41 +548,42 @@ public boolean supportsGetGeneratedKeys() throws MetaException {
     }
   }
 
-  public boolean isDuplicateKeyError(SQLException ex) {
+  public boolean isDuplicateKeyError(Throwable ex) {

Review Comment:
   minor, i would just change input param to `t`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418921836


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ProgrammaticRollbackException.java:
##########
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.txn.jdbc;
+
+import java.lang.reflect.Method;
+
+/**
+ * This exception can be used to trigger rollback in 
+ * {@link org.apache.hadoop.hive.metastore.txn.TransactionalRetryProxy#invoke(Object, Method, Object[])}
+ * for the current transaction, without propagating the exception to the caller. The proxy will catch this exception,
+ * rollback the transaction (if not yet completed already) and return the value supplied in the constructor to te caller.
+ */
+public class ProgrammaticRollbackException extends RuntimeException {  

Review Comment:
   can't we simply call it RollbackException?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1418927929


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/retryhandling/SqlRetryHandler.java:
##########
@@ -123,32 +124,29 @@ public <Result> Result executeWithRetry(SqlRetryCallProperties properties, SqlRe
       if (properties.isLockInternally()) {
         lockInternal();
       }
-      threadLocal.set(new ContextNode<>(threadLocal.get(), new Object()));
+      threadLocal.set(new Object());

Review Comment:
   what does it do?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "veghlaci05 (via GitHub)" <gi...@apache.org>.
veghlaci05 commented on code in PR #4566:
URL: https://github.com/apache/hive/pull/4566#discussion_r1399121112


##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/jdbc/ParameterizedBatchCommand.java:
##########
@@ -0,0 +1,61 @@
+/*

Review Comment:
   I was thinking the other way around: These interfaces and abstract classes are parameters in the various overloads of `MultiDataSourceJdbcResource.execute()` methods. Therefore I thought these should be in the same package as these classes together form the "API" to use by `[Compaction]TxnHandler` function implementations. However, I'm not strictly tied to this idea, as it doesn't really matter in which of these packages we put them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27481: TxnHandler cleanup [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1835731226

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![D](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/D-16px.png 'D')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [18 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [168 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] sonarcloud[bot] commented on pull request #4566: HIVE-27481: TxnHandler cleanup

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4566:
URL: https://github.com/apache/hive/pull/4566#issuecomment-1682058614

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4566)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [![C](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C-16px.png 'C')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG) [8 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4566&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL) [130 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4566&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4566&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org