You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/06/22 15:22:22 UTC

[GitHub] [hive] pvargacl commented on a change in pull request #1151: HIVE-23725: ValidTxnManager snapshot outdating causing partial reads

pvargacl commented on a change in pull request #1151:
URL: https://github.com/apache/hive/pull/1151#discussion_r443639126



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/Driver.java
##########
@@ -675,50 +678,18 @@ private void runInternal(String command, boolean alreadyCompiled) throws Command
 
       try {
         if (!validTxnManager.isValidTxnListState()) {
-          LOG.info("Compiling after acquiring locks");
+          LOG.info("Reexecuting after acquiring locks, since snapshot was outdated.");
           // Snapshot was outdated when locks were acquired, hence regenerate context,
-          // txn list and retry
-          // TODO: Lock acquisition should be moved before analyze, this is a bit hackish.
-          // Currently, we acquire a snapshot, we compile the query wrt that snapshot,
-          // and then, we acquire locks. If snapshot is still valid, we continue as usual.
-          // But if snapshot is not valid, we recompile the query.
-          if (driverContext.isOutdatedTxn()) {
-            driverContext.getTxnManager().rollbackTxn();
-
-            String userFromUGI = DriverUtils.getUserFromUGI(driverContext);
-            driverContext.getTxnManager().openTxn(context, userFromUGI, driverContext.getTxnType());
-            lockAndRespond();
-          }
-          driverContext.setRetrial(true);
-          driverContext.getBackupContext().addSubContext(context);
-          driverContext.getBackupContext().setHiveLocks(context.getHiveLocks());
-          context = driverContext.getBackupContext();
-          driverContext.getConf().set(ValidTxnList.VALID_TXNS_KEY,
-            driverContext.getTxnManager().getValidTxns().toString());
-          if (driverContext.getPlan().hasAcidResourcesInQuery()) {
-            validTxnManager.recordValidWriteIds();
-          }
-
-          if (!alreadyCompiled) {
-            // compile internal will automatically reset the perf logger
-            compileInternal(command, true);
-          } else {
-            // Since we're reusing the compiled plan, we need to update its start time for current run
-            driverContext.getPlan().setQueryStartTime(driverContext.getQueryDisplay().getQueryStartTime());
-          }
-
-          if (!validTxnManager.isValidTxnListState()) {
-            // Throw exception
-            throw handleHiveException(new HiveException("Operation could not be executed"), 14);
+          // txn list and retry (see ReExecutionRetryLockPlugin)
+          try {
+            releaseLocksAndCommitOrRollback(false);
+          } catch (LockException e) {
+            handleHiveException(e, 12);
           }
-
-          //Reset the PerfLogger
-          perfLogger = SessionState.getPerfLogger(true);
-
-          // the reason that we set the txn manager for the cxt here is because each
-          // query has its own ctx object. The txn mgr is shared across the
-          // same instance of Driver, which can run multiple queries.
-          context.setHiveTxnManager(driverContext.getTxnManager());
+          throw handleHiveException(

Review comment:
       This is an interesting question. In the original logic, if an other commit invalidated the snaphsot a second time, the query also failed with HiveExection. The main difference is, we do more work in this case (compile and acquire the locks again), so the chance is probably higher that the snapshot gets invalidated a second time, but I don't know if it is high enough that we should consider it. The ReexecDriver uses one global config for the number of retries, it would take some refactoring to make it independently configurable for the different plugins.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org