You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by GitBox <gi...@apache.org> on 2022/02/04 21:46:20 UTC

[GitHub] [accumulo] dlmarion opened a new pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

dlmarion opened a new pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467


   This change adds a new Thrift API method to the FateServiceHandler to
   cancel an existing FaTE transaction. The API method is called from
   FateCommand using the ManagerClient class. This will attempt to set
   the status to failed for a non-running transaction and will interrupt
   the TransactionRunner thread for a transaction that is currently in
   progress
   
   Closes #2328 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] EdColeman commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800780155



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       I do not know if there is a nuance with cancel vs. fail.  Cancel seems (to me) a more normal operation - I changed my mind. Fail seems more of an error condition.  The documentation in the FATE section states that  "Failing an operation is not a normal procedure" Does canceling a compaction raise to that level?  So either way fail / or cancel, but it maybe worth considering the context and usage.   
   
   One one hand, not needing to remember when its fail vs. cancel (follow existing naming) is a positive - unless cancel is more "normal" and does not take a lot of insight for an operator to execute.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801847709



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,24 +83,51 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
+            boolean exceptionSeen = false;
             try {
-              deferTime = op.isReady(tid, environment);
-
-              if (deferTime == 0) {
-                prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+              try {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());

Review comment:
       ```suggestion
                     // If this ever fails its indication that there is a concurrency problem in other code because its not  expected that multiple threads would be working on the same transaction id at the same time.
                     Preconditions.checkState(RUNNING_TRANSACTIONS.putIfAbsent(tid,Thread.currentThread()) == null);
   ```

##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,24 +83,51 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
+            boolean exceptionSeen = false;
             try {
-              deferTime = op.isReady(tid, environment);
-
-              if (deferTime == 0) {
-                prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+              try {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
                 }
-                op = op.call(tid, environment);
-              } else
-                continue;
-
+                deferTime = op.isReady(tid, environment);
+                if (deferTime == 0) {
+                  prevOp = op;
+                  if (status == SUBMITTED) {
+                    store.setStatus(tid, IN_PROGRESS);
+                  }
+                  op = op.call(tid, environment);
+                } else
+                  continue;
+              } catch (Exception e) {
+                exceptionSeen = true;
+                throw e;
+              } finally {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.remove(tid);

Review comment:
       ```suggestion
                     // unexpected that the following should fail, if it does probably indicates there is a semantic error in the code
                     Preconditions.checkState(RUNNING_TRANSACTIONS.remove(tid) != null);
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800900299



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,35 +83,58 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
+              synchronized (RUNNING_TRANSACTIONS) {
+                RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
+              }
               deferTime = op.isReady(tid, environment);
 
               if (deferTime == 0) {
                 prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+                if (status == SUBMITTED) {
+                  store.setStatus(tid, IN_PROGRESS);
                 }
                 op = op.call(tid, environment);
               } else
                 continue;
 
             } catch (Exception e) {
+              if (e instanceof InterruptedException) {
+                runnerLog.info(
+                    "FATE Runner thread interrupted processing txid " + Long.toHexString(tid));
+                if (prevOp != null) {
+                  processFailed(tid, prevOp);
+                }
+                if (op != prevOp) {
+                  processFailed(tid, op);
+                }
+                Thread.interrupted();

Review comment:
       How so? I was using this method to clear the interrupted state of the current Thread. https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Thread.html#interrupted()




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801753003



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       > It would also be desirable that a delete operation take precedence over any other operation in progress or queued for a table. If a compaction or a bulk import is running and then the system requests the table be deleted ...
   
   Delete table cancels users compactions now https://github.com/apache/accumulo/pull/2169




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800837469



##########
File path: test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
##########
@@ -175,6 +205,170 @@ public void testTransactionStatus() throws Exception {
     }
   }
 
+  @Test
+  public void testCancelWhileNew() throws Exception {
+    final ZooReaderWriter zk = new ZooReaderWriter(szk.getConn(), 30000, "secret");
+    final ZooStore<Manager> zooStore = new ZooStore<Manager>(ZK_ROOT + Constants.ZFATE, zk);
+    final AgeOffStore<Manager> store = new AgeOffStore<Manager>(zooStore, 3000);
+
+    Manager manager = createMock(Manager.class);
+    ServerContext sctx = createMock(ServerContext.class);
+    expect(manager.getContext()).andReturn(sctx).anyTimes();
+    expect(sctx.getZooKeeperRoot()).andReturn(ZK_ROOT).anyTimes();
+    expect(sctx.getZooReaderWriter()).andReturn(zk).anyTimes();
+    replay(manager, sctx);
+
+    Fate<Manager> fate = new Fate<Manager>(manager, store, TraceRepo::toLogString);
+    try {
+      ConfigurationCopy config = new ConfigurationCopy();
+      config.set(Property.GENERAL_SIMPLETIMER_THREADPOOL_SIZE, "2");
+      config.set(Property.MANAGER_FATE_THREADPOOL_SIZE, "1");
+      fate.startTransactionRunners(config);
+
+      // Wait for the transaction runner to be scheduled.
+      UtilWaitThread.sleep(3000);
+
+      doReady = new CountDownLatch(1);
+      callStarted = new CountDownLatch(1);
+      finishCall = new CountDownLatch(1);
+
+      long txid = fate.startTransaction();
+      LOG.info("Starting test testCancelWhileNew with {}", Long.toHexString(txid));
+      assertEquals(TStatus.NEW, getTxStatus(zk, txid));
+      // cancel the transaction
+      assertTrue(fate.cancel(txid));
+      assertTrue(TStatus.FAILED_IN_PROGRESS == getTxStatus(zk, txid)
+          || TStatus.FAILED == getTxStatus(zk, txid));
+      fate.seedTransaction(txid, new TestOperation(NS, TID), true);
+      assertTrue(TStatus.FAILED_IN_PROGRESS == getTxStatus(zk, txid)
+          || TStatus.FAILED == getTxStatus(zk, txid));
+    } finally {
+      fate.shutdown();
+    }
+  }
+
+  @Test
+  public void testCancelWhileSubmitted() throws Exception {
+    final ZooReaderWriter zk = new ZooReaderWriter(szk.getConn(), 30000, "secret");
+    final ZooStore<Manager> zooStore = new ZooStore<Manager>(ZK_ROOT + Constants.ZFATE, zk);
+    final AgeOffStore<Manager> store = new AgeOffStore<Manager>(zooStore, 3000);
+
+    Manager manager = createMock(Manager.class);
+    ServerContext sctx = createMock(ServerContext.class);
+    expect(manager.getContext()).andReturn(sctx).anyTimes();
+    expect(sctx.getZooKeeperRoot()).andReturn(ZK_ROOT).anyTimes();
+    expect(sctx.getZooReaderWriter()).andReturn(zk).anyTimes();
+    replay(manager, sctx);
+
+    Fate<Manager> fate = new Fate<Manager>(manager, store, TraceRepo::toLogString);
+    try {
+      ConfigurationCopy config = new ConfigurationCopy();
+      config.set(Property.GENERAL_SIMPLETIMER_THREADPOOL_SIZE, "2");
+      config.set(Property.MANAGER_FATE_THREADPOOL_SIZE, "1");
+      fate.startTransactionRunners(config);
+
+      // Wait for the transaction runner to be scheduled.
+      UtilWaitThread.sleep(3000);
+
+      doReady = new CountDownLatch(1);
+      callStarted = new CountDownLatch(1);
+      finishCall = new CountDownLatch(1);
+
+      long txid = fate.startTransaction();
+      LOG.info("Starting test testCancelWhileSubmitted with {}", Long.toHexString(txid));
+      assertEquals(TStatus.NEW, getTxStatus(zk, txid));
+      fate.seedTransaction(txid, new TestOperation(NS, TID), true);
+      assertEquals(TStatus.SUBMITTED, getTxStatus(zk, txid));
+      // cancel the transaction
+      assertTrue(fate.cancel(txid));
+      // do the isReady method
+      doReady.countDown();
+      // Check that tx is failing or gets removed
+      boolean nodeRemoved = false;
+      while (!nodeRemoved) {
+        try {
+          TStatus s = getTxStatus(zk, txid);
+          assertTrue(
+              s == TStatus.SUBMITTED || s == TStatus.FAILED_IN_PROGRESS || s == TStatus.FAILED);

Review comment:
       Addressed in b87b8f1




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800933167



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,35 +83,58 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
+              synchronized (RUNNING_TRANSACTIONS) {
+                RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
+              }
               deferTime = op.isReady(tid, environment);
 
               if (deferTime == 0) {
                 prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+                if (status == SUBMITTED) {
+                  store.setStatus(tid, IN_PROGRESS);
                 }
                 op = op.call(tid, environment);
               } else
                 continue;
 
             } catch (Exception e) {
+              if (e instanceof InterruptedException) {
+                runnerLog.info(
+                    "FATE Runner thread interrupted processing txid " + Long.toHexString(tid));
+                if (prevOp != null) {
+                  processFailed(tid, prevOp);
+                }
+                if (op != prevOp) {
+                  processFailed(tid, op);
+                }

Review comment:
       Ok, I see. Yeah, I can remove those.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800989448



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       To Ed's question, "Does canceling a compaction raise to that level?" ... I think both operations are pretty abnormal... to the point that I'm still wondering whether we really need to be able to do this live when taking the manager offline is probably fine, and safest, under abnormal conditions. I worry that people will normalize it, and their systems will become unstable over time by abusing this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r802197735



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       > Feedback from user(s) is:
   > 
   > 2. Stopping the Manager is a problem because they can't load data via bulk imports, they have strict timeliness requirements. They have to stop bulk imports from getting kicked off, wait for the current ones to finish, then take down the Manager. Stopping the Manager also stops any long-running WAL recovery due to a failed TServer, table balancing, etc.
   
   I understand stopping the manager is not preferred. I'm not sure lowering the bar to surgery by allowing it to be performed on a conscious patient is the right solution, though (might be taking the metaphor too far here :smiley_cat: ). I'd rather focus efforts on surgery-prevention, if we can. For example, using the new bulk imports, using the new working compaction queue cancellation, figuring out why 1.x compactions can't be cancelled, etc., would be a great start. Adding range locks for compaction/merge/delete would also be great.
   
   Right now, this seems very risky to add, especially since we haven't even ironed out the kinks in the other recent changes in these areas. ExternalCompactions ITs are still flaky, and now FateIT is failing too. These imply that there is instability in this code. Giving users more direct access to making live changes to internals, when we haven't ironed out those internals in our own tests, seems very high risk, and the wrong solution to the underlying problems users are seeing. If all those other things were working well, I would be less opposed to taking the risk here, but they're not. So, this risk just seems too much right now.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800822471



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       SInce I equated `fail` with `cancel` I tried to use a name that implied the user could cancel/fail the operation with the manager up. Hence `fail-live`. That sounded better than `fail-up`, lol.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800885134



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +335,55 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          t.interrupt();

Review comment:
       There is a race condition here.  The same thread runs many different FATE tx steps.  By the time this code executes, the thread could be running a completely different FATE tx,  so it could interrupt the wrong FATE tx.

##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,35 +83,58 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
+              synchronized (RUNNING_TRANSACTIONS) {
+                RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
+              }
               deferTime = op.isReady(tid, environment);
 
               if (deferTime == 0) {
                 prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+                if (status == SUBMITTED) {
+                  store.setStatus(tid, IN_PROGRESS);
                 }
                 op = op.call(tid, environment);
               } else
                 continue;
 
             } catch (Exception e) {
+              if (e instanceof InterruptedException) {
+                runnerLog.info(
+                    "FATE Runner thread interrupted processing txid " + Long.toHexString(tid));
+                if (prevOp != null) {
+                  processFailed(tid, prevOp);
+                }
+                if (op != prevOp) {
+                  processFailed(tid, op);
+                }

Review comment:
       This code calls processFailed before transitioning the fate tx to FAILED_IN_PROGRESS.  If the manager process dies in the middle of processFailed() then when it comes back it will start executing the tx in the IN_PROGRESS state which could lead to normal processing, failure processing, followed by normal processing for a FATE tx.   This could lead to strange mutations to ZK and metadata table. 
   
   What is the motivation for this special case? 

##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,35 +83,58 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
+              synchronized (RUNNING_TRANSACTIONS) {
+                RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
+              }
               deferTime = op.isReady(tid, environment);
 
               if (deferTime == 0) {
                 prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+                if (status == SUBMITTED) {
+                  store.setStatus(tid, IN_PROGRESS);
                 }
                 op = op.call(tid, environment);
               } else
                 continue;
 
             } catch (Exception e) {
+              if (e instanceof InterruptedException) {
+                runnerLog.info(
+                    "FATE Runner thread interrupted processing txid " + Long.toHexString(tid));
+                if (prevOp != null) {
+                  processFailed(tid, prevOp);
+                }
+                if (op != prevOp) {
+                  processFailed(tid, op);
+                }
+                Thread.interrupted();

Review comment:
       Not sure, but this may prevent the following code from running.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800837469



##########
File path: test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
##########
@@ -175,6 +205,170 @@ public void testTransactionStatus() throws Exception {
     }
   }
 
+  @Test
+  public void testCancelWhileNew() throws Exception {
+    final ZooReaderWriter zk = new ZooReaderWriter(szk.getConn(), 30000, "secret");
+    final ZooStore<Manager> zooStore = new ZooStore<Manager>(ZK_ROOT + Constants.ZFATE, zk);
+    final AgeOffStore<Manager> store = new AgeOffStore<Manager>(zooStore, 3000);
+
+    Manager manager = createMock(Manager.class);
+    ServerContext sctx = createMock(ServerContext.class);
+    expect(manager.getContext()).andReturn(sctx).anyTimes();
+    expect(sctx.getZooKeeperRoot()).andReturn(ZK_ROOT).anyTimes();
+    expect(sctx.getZooReaderWriter()).andReturn(zk).anyTimes();
+    replay(manager, sctx);
+
+    Fate<Manager> fate = new Fate<Manager>(manager, store, TraceRepo::toLogString);
+    try {
+      ConfigurationCopy config = new ConfigurationCopy();
+      config.set(Property.GENERAL_SIMPLETIMER_THREADPOOL_SIZE, "2");
+      config.set(Property.MANAGER_FATE_THREADPOOL_SIZE, "1");
+      fate.startTransactionRunners(config);
+
+      // Wait for the transaction runner to be scheduled.
+      UtilWaitThread.sleep(3000);
+
+      doReady = new CountDownLatch(1);
+      callStarted = new CountDownLatch(1);
+      finishCall = new CountDownLatch(1);
+
+      long txid = fate.startTransaction();
+      LOG.info("Starting test testCancelWhileNew with {}", Long.toHexString(txid));
+      assertEquals(TStatus.NEW, getTxStatus(zk, txid));
+      // cancel the transaction
+      assertTrue(fate.cancel(txid));
+      assertTrue(TStatus.FAILED_IN_PROGRESS == getTxStatus(zk, txid)
+          || TStatus.FAILED == getTxStatus(zk, txid));
+      fate.seedTransaction(txid, new TestOperation(NS, TID), true);
+      assertTrue(TStatus.FAILED_IN_PROGRESS == getTxStatus(zk, txid)
+          || TStatus.FAILED == getTxStatus(zk, txid));
+    } finally {
+      fate.shutdown();
+    }
+  }
+
+  @Test
+  public void testCancelWhileSubmitted() throws Exception {
+    final ZooReaderWriter zk = new ZooReaderWriter(szk.getConn(), 30000, "secret");
+    final ZooStore<Manager> zooStore = new ZooStore<Manager>(ZK_ROOT + Constants.ZFATE, zk);
+    final AgeOffStore<Manager> store = new AgeOffStore<Manager>(zooStore, 3000);
+
+    Manager manager = createMock(Manager.class);
+    ServerContext sctx = createMock(ServerContext.class);
+    expect(manager.getContext()).andReturn(sctx).anyTimes();
+    expect(sctx.getZooKeeperRoot()).andReturn(ZK_ROOT).anyTimes();
+    expect(sctx.getZooReaderWriter()).andReturn(zk).anyTimes();
+    replay(manager, sctx);
+
+    Fate<Manager> fate = new Fate<Manager>(manager, store, TraceRepo::toLogString);
+    try {
+      ConfigurationCopy config = new ConfigurationCopy();
+      config.set(Property.GENERAL_SIMPLETIMER_THREADPOOL_SIZE, "2");
+      config.set(Property.MANAGER_FATE_THREADPOOL_SIZE, "1");
+      fate.startTransactionRunners(config);
+
+      // Wait for the transaction runner to be scheduled.
+      UtilWaitThread.sleep(3000);
+
+      doReady = new CountDownLatch(1);
+      callStarted = new CountDownLatch(1);
+      finishCall = new CountDownLatch(1);
+
+      long txid = fate.startTransaction();
+      LOG.info("Starting test testCancelWhileSubmitted with {}", Long.toHexString(txid));
+      assertEquals(TStatus.NEW, getTxStatus(zk, txid));
+      fate.seedTransaction(txid, new TestOperation(NS, TID), true);
+      assertEquals(TStatus.SUBMITTED, getTxStatus(zk, txid));
+      // cancel the transaction
+      assertTrue(fate.cancel(txid));
+      // do the isReady method
+      doReady.countDown();
+      // Check that tx is failing or gets removed
+      boolean nodeRemoved = false;
+      while (!nodeRemoved) {
+        try {
+          TStatus s = getTxStatus(zk, txid);
+          assertTrue(
+              s == TStatus.SUBMITTED || s == TStatus.FAILED_IN_PROGRESS || s == TStatus.FAILED);

Review comment:
       Addressed in c9e64f4




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801040221



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +335,55 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          t.interrupt();

Review comment:
       > I re-worked the synchronization in [6d22ea7](https://github.com/apache/accumulo/pull/2467/commits/6d22ea7708b786a9b9cd2aa4469e332e29e57476)
   
   That may be a good solution to the problem.  I suggested some tweaks to the approach.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801875171



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       Adding ranges to the tables locks that FATE acquires seems like a good thing to explore.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r802669973



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       > > might be taking the metaphor too far here
   > 
   > I'm not convinced that it's the right one as it's [documented](https://accumulo.apache.org/docs/2.x/administration/fate). For me, surgery is manually inserting / updating / deleting rows from the metadata table.
   
   Internals can still be documented. We document the metadata structure and have shared knowledge in various forms on how to do that kind of surgery as well. The key point for me is that FaTE operations don't have a public API. It is an implementation detail of other operations. In any case, if it helps, I'm happy to concede that surgery is not a great metaphor, but I do think it's worth noting that users should never have to touch with it under normal operations.
   
   > 
   > > Right now, this seems very risky to add, especially since we haven't even ironed out the kinks in the other recent changes in these areas. ExternalCompactions ITs are still flaky, and now FateIT is failing too.
   > 
   > I"m starting to think the issue is ZK based on the information you added to in #2463 and the fact that FateIT has one test.
   
   Perhaps. More troubleshooting needs to be done for those, I think, to draw a sound conclusion. The relevance for this PR, though, is that I would like to hold off on introducing more high-risk, potentially destabilizing code, until the reliability of what's there now is improved.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801026209



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       So, we give users the ability to create fate transactions easily. I think in only one case, we give them the ability to cancel them (cancel compactions). Taking the manager down to fail a fate transaction is very disruptive (and doesn't it terminate the FaTE transaction runners in the middle of their operations?). Certainly the changes I made in #2462 will let users identify which transactions have never been started, those should be able to be cancelled with no issue.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800980619



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,35 +83,58 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
+              synchronized (RUNNING_TRANSACTIONS) {
+                RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
+              }
               deferTime = op.isReady(tid, environment);
 
               if (deferTime == 0) {
                 prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+                if (status == SUBMITTED) {
+                  store.setStatus(tid, IN_PROGRESS);
                 }
                 op = op.call(tid, environment);
               } else
                 continue;
 
             } catch (Exception e) {
+              if (e instanceof InterruptedException) {
+                runnerLog.info(
+                    "FATE Runner thread interrupted processing txid " + Long.toHexString(tid));
+                if (prevOp != null) {
+                  processFailed(tid, prevOp);
+                }
+                if (op != prevOp) {
+                  processFailed(tid, op);
+                }

Review comment:
       Removed in [6d22ea7](https://github.com/apache/accumulo/pull/2467/commits/6d22ea7708b786a9b9cd2aa4469e332e29e57476)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801090182



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +331,68 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner. Look up
+        // the thread that is running this transaction.
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          Thread t2 = null;
+          synchronized (t) {
+            // Make sure that the transaction is still associated
+            // with this thread. If it is, then this thread is blocked
+            // from removing this transaction from the table and we can
+            // interrupt it and cause it to fail.
+            synchronized (RUNNING_TRANSACTIONS) {
+              t2 = RUNNING_TRANSACTIONS.get(tid);
+            }
+            if (t2 != null && t == t2) {
+              t.interrupt();
+            }
+          }
+          return true;
+        } else {
+          // reserved, but not in transaction table. It should transition to
+          // an end state or become unreserved in a short amount of time.
+          // Lets retry.
+          UtilWaitThread.sleep(500);
+        }

Review comment:
       Incorporated this change into [c49f55e](https://github.com/apache/accumulo/pull/2467/commits/c49f55ec90ed3ea883c9dbfd22dfac740b3e8b3d)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801036570



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +331,68 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner. Look up
+        // the thread that is running this transaction.
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          Thread t2 = null;
+          synchronized (t) {
+            // Make sure that the transaction is still associated
+            // with this thread. If it is, then this thread is blocked
+            // from removing this transaction from the table and we can
+            // interrupt it and cause it to fail.
+            synchronized (RUNNING_TRANSACTIONS) {
+              t2 = RUNNING_TRANSACTIONS.get(tid);
+            }
+            if (t2 != null && t == t2) {
+              t.interrupt();
+            }
+          }
+          return true;
+        } else {
+          // reserved, but not in transaction table. It should transition to
+          // an end state or become unreserved in a short amount of time.
+          // Lets retry.
+          UtilWaitThread.sleep(500);
+        }

Review comment:
       ```suggestion
           Thread t = null;
           synchronized (RUNNING_TRANSACTIONS) {
             t = RUNNING_TRANSACTIONS.get(tid);
             if( t != null){
                //must interrupt in sync block to ensure thread is still associated with transaction... 
                // however if we interrupt the thread and nothing checks the interrupt status then 
                // it could still impact an unrelated FATE tx, added a TODO about that elsewhere
                t.interrupt();
                return true; 
             }
           }
           
             // TODO not sure if I got the flow/branch on the following part correct
                       
             // reserved, but not in transaction table. It should transition to
             // an end state or become unreserved in a short amount of time.
             // Lets retry.
             UtilWaitThread.sleep(500);
   ```

##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,24 +83,43 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
-              deferTime = op.isReady(tid, environment);
-
-              if (deferTime == 0) {
-                prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+              try {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
                 }
-                op = op.call(tid, environment);
-              } else
-                continue;
-
+                deferTime = op.isReady(tid, environment);
+                if (deferTime == 0) {
+                  prevOp = op;
+                  if (status == SUBMITTED) {
+                    store.setStatus(tid, IN_PROGRESS);
+                  }
+                  op = op.call(tid, environment);
+                } else
+                  continue;
+              } finally {
+                synchronized (Thread.currentThread()) {
+                  synchronized (RUNNING_TRANSACTIONS) {
+                    RUNNING_TRANSACTIONS.remove(tid);
+                  }
+                }

Review comment:
       ```suggestion
                     synchronized (RUNNING_TRANSACTIONS) {
                       RUNNING_TRANSACTIONS.remove(tid);
                     }
                     // TODO maybe we should check/clear for interrupt here... if something 
                     // was trying to interrupt right before the sync block then we do not want 
                     // that to impact the next thing this thread will work on....
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801197041



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,24 +83,51 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
+            boolean ieSeen = false;
             try {
-              deferTime = op.isReady(tid, environment);
-
-              if (deferTime == 0) {
-                prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+              try {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
                 }
-                op = op.call(tid, environment);
-              } else
-                continue;
-
+                deferTime = op.isReady(tid, environment);
+                if (deferTime == 0) {
+                  prevOp = op;
+                  if (status == SUBMITTED) {
+                    store.setStatus(tid, IN_PROGRESS);
+                  }
+                  op = op.call(tid, environment);
+                } else
+                  continue;
+              } catch (InterruptedException ie) {
+                ieSeen = true;
+                throw ie;
+              } finally {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.remove(tid);
+                }
+                // It's possible that cancel was called between
+                // op.call and the synchronized block above. Do a last
+                // check to see if that's the case.
+                if (!ieSeen && Thread.currentThread().isInterrupted()) {
+                  throw new InterruptedException("FaTE thread was interrupted");
+                }

Review comment:
       If there is any exception on the try{} block then we probably do not want to throw anything in the finally block as it will obscure the original exception.  Generalizing the code as follows may avoid this problem.
   
   ```suggestion
                 } catch (Exception e) {
                   eSeen = true;
                   throw e;
                 } finally {
                   synchronized (RUNNING_TRANSACTIONS) {
                     RUNNING_TRANSACTIONS.remove(tid);
                   }
                   // It's possible that cancel was called between
                   // op.call and the synchronized block above. Do a last
                   // check to see if that's the case.
                   if (!eSeen && Thread.currentThread().isInterrupted()) {
                     throw new InterruptedException("FaTE thread was interrupted");
                   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#issuecomment-1031682906


   @keith-turner reached out about my change in #2462 to highlight the fact that even after my change an IN_PROGRESS transaction may not be currently running as the status is set to IN_PROGRESS when the TransactionRunner deals with the first op in a fate transaction op chain. The subsequent op gets pushed back into ZK to be operated on at a later point. I believe that this change will still cancel a Fate Tx in this scenario as the tx will not be reserved so `Fate.cancel()` will set the status to failed in progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] milleruntime commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800764709



##########
File path: test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
##########
@@ -175,6 +205,170 @@ public void testTransactionStatus() throws Exception {
     }
   }
 
+  @Test
+  public void testCancelWhileNew() throws Exception {
+    final ZooReaderWriter zk = new ZooReaderWriter(szk.getConn(), 30000, "secret");
+    final ZooStore<Manager> zooStore = new ZooStore<Manager>(ZK_ROOT + Constants.ZFATE, zk);
+    final AgeOffStore<Manager> store = new AgeOffStore<Manager>(zooStore, 3000);
+
+    Manager manager = createMock(Manager.class);
+    ServerContext sctx = createMock(ServerContext.class);
+    expect(manager.getContext()).andReturn(sctx).anyTimes();
+    expect(sctx.getZooKeeperRoot()).andReturn(ZK_ROOT).anyTimes();
+    expect(sctx.getZooReaderWriter()).andReturn(zk).anyTimes();
+    replay(manager, sctx);
+
+    Fate<Manager> fate = new Fate<Manager>(manager, store, TraceRepo::toLogString);
+    try {
+      ConfigurationCopy config = new ConfigurationCopy();
+      config.set(Property.GENERAL_SIMPLETIMER_THREADPOOL_SIZE, "2");
+      config.set(Property.MANAGER_FATE_THREADPOOL_SIZE, "1");
+      fate.startTransactionRunners(config);
+
+      // Wait for the transaction runner to be scheduled.
+      UtilWaitThread.sleep(3000);
+
+      doReady = new CountDownLatch(1);
+      callStarted = new CountDownLatch(1);
+      finishCall = new CountDownLatch(1);
+
+      long txid = fate.startTransaction();
+      LOG.info("Starting test testCancelWhileNew with {}", Long.toHexString(txid));
+      assertEquals(TStatus.NEW, getTxStatus(zk, txid));
+      // cancel the transaction
+      assertTrue(fate.cancel(txid));
+      assertTrue(TStatus.FAILED_IN_PROGRESS == getTxStatus(zk, txid)
+          || TStatus.FAILED == getTxStatus(zk, txid));
+      fate.seedTransaction(txid, new TestOperation(NS, TID), true);
+      assertTrue(TStatus.FAILED_IN_PROGRESS == getTxStatus(zk, txid)
+          || TStatus.FAILED == getTxStatus(zk, txid));
+    } finally {
+      fate.shutdown();
+    }
+  }
+
+  @Test
+  public void testCancelWhileSubmitted() throws Exception {
+    final ZooReaderWriter zk = new ZooReaderWriter(szk.getConn(), 30000, "secret");
+    final ZooStore<Manager> zooStore = new ZooStore<Manager>(ZK_ROOT + Constants.ZFATE, zk);
+    final AgeOffStore<Manager> store = new AgeOffStore<Manager>(zooStore, 3000);
+
+    Manager manager = createMock(Manager.class);
+    ServerContext sctx = createMock(ServerContext.class);
+    expect(manager.getContext()).andReturn(sctx).anyTimes();
+    expect(sctx.getZooKeeperRoot()).andReturn(ZK_ROOT).anyTimes();
+    expect(sctx.getZooReaderWriter()).andReturn(zk).anyTimes();
+    replay(manager, sctx);
+
+    Fate<Manager> fate = new Fate<Manager>(manager, store, TraceRepo::toLogString);
+    try {
+      ConfigurationCopy config = new ConfigurationCopy();
+      config.set(Property.GENERAL_SIMPLETIMER_THREADPOOL_SIZE, "2");
+      config.set(Property.MANAGER_FATE_THREADPOOL_SIZE, "1");
+      fate.startTransactionRunners(config);
+
+      // Wait for the transaction runner to be scheduled.
+      UtilWaitThread.sleep(3000);
+
+      doReady = new CountDownLatch(1);
+      callStarted = new CountDownLatch(1);
+      finishCall = new CountDownLatch(1);
+
+      long txid = fate.startTransaction();
+      LOG.info("Starting test testCancelWhileSubmitted with {}", Long.toHexString(txid));
+      assertEquals(TStatus.NEW, getTxStatus(zk, txid));
+      fate.seedTransaction(txid, new TestOperation(NS, TID), true);
+      assertEquals(TStatus.SUBMITTED, getTxStatus(zk, txid));
+      // cancel the transaction
+      assertTrue(fate.cancel(txid));
+      // do the isReady method
+      doReady.countDown();
+      // Check that tx is failing or gets removed
+      boolean nodeRemoved = false;
+      while (!nodeRemoved) {
+        try {
+          TStatus s = getTxStatus(zk, txid);
+          assertTrue(
+              s == TStatus.SUBMITTED || s == TStatus.FAILED_IN_PROGRESS || s == TStatus.FAILED);

Review comment:
       Static imports could clean up the formatting here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800938509



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +335,55 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          t.interrupt();

Review comment:
       I might also need to move the RUNNING_TRANSACTIONS.remove(tid) up a little and tighten that up.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] milleruntime commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800648662



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +328,56 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == TStatus.NEW || status == TStatus.SUBMITTED
+              || status == TStatus.IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(

Review comment:
       What is `EXCEPTION_PROP`? I can't seem to find it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800903110



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +335,55 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          t.interrupt();

Review comment:
       This problem seems tricky.  One possible way to work around it is to create new thread for running each FATE step.  Then its always safe to interrupt it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801174267



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       @dlmarion wrote:
   > So, we give users the ability to create fate transactions easily.
   
   I don't think that's an accurate characterization. Users don't create fate transactions at all. The Manager does so to service some user requests, like "create a table". This distinction matters, because fate transactions are an implementation detail and should largely be hidden from the user. In many cases, these operations are expected to be so fast (like creating a table), the user shouldn't even have the concept of it being a queued action. The only reason we run them as a fate operation is so we don't leave Accumulo in a dirty state in the case of a failure. In the create table example, users don't need to be able to cancel the fate operations, because they can just delete an unwanted table.
   
   So, in a lot of cases, the fate operations aren't actually there for the user. They are there for Accumulo's internal needs. Implementation detail. So, it makes a kind of sense that users don't directly cancel them, or can't, under normal circumstances.
   
   >  I think in only one case, we give them the ability to cancel them (cancel compactions).
   
   The reason users can easily cancel compaction is because the semantics for the compaction API is a "queue this compaction" request, not an "execute this compaction immediately" request. The cancellation for compactions is available to users because of the semantics of the compaction API, not because of the implementation detail of it using fate to execute it. Those queuing and cancelling semantics aren't generalizable to all actions that we perform via fate and should be considered on a case-by-case basis and exposed through our normal APIs, just like compactions are today. Interacting with Fate is not going through normal APIs. It's surgery, and should be rare.
   
   > Taking the manager down to fail a fate transaction is very disruptive (and doesn't it terminate the FaTE transaction runners in the middle of their operations?).
   
   Aborting transaction runners in the middle of an operation is generally safe because FaTE RepOs are designed to be idempotent and replayable to resume a transaction. They also have the ability to roll back when they are move into a failed state. The reason we kill the manager is to ensure we know the current state of any transactions that are in progress that we intend to fail, so there aren't any race conditions or risks of some side-effect of a poorly behaved interrupted thread. That's generally acceptable because aborting fate operations is supposed to be rare, and the surgery should be relatively safe, with predictable outcomes. But, it's still surgery, and we don't exactly want to lower the barrier to entry to surgery.
   
   Fate is not intended to be a general request queue that users add and remove stuff from at will, they are for internal consistency. When we fail stuff, it's because we're trying to force something that isn't normal. If we want some operations to be cancel-able, I think maybe creating a dedicated asynchronous API for the operations we want to be able to cancel might be a better approach, than trying to lower the barrier to entry to Fate surgery or trying to generalize Fate operations like it's a general purpose executor service.
   
   > Certainly the changes I made in #2462 will let users identify which transactions have never been started, those should be able to be cancelled with no issue.
   
   Is the intent of this current PR to only be able to `fail-live` things that have never been started, or operations in any state?  If it's only to help clear out unstarted things, I think I'd be more enthusiastic about it. It might even be useful have a safe live option for `cancel all transactions not yet started`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801757076



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,24 +83,51 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
+            boolean ieSeen = false;
             try {
-              deferTime = op.isReady(tid, environment);
-
-              if (deferTime == 0) {
-                prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+              try {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
                 }
-                op = op.call(tid, environment);
-              } else
-                continue;
-
+                deferTime = op.isReady(tid, environment);
+                if (deferTime == 0) {
+                  prevOp = op;
+                  if (status == SUBMITTED) {
+                    store.setStatus(tid, IN_PROGRESS);
+                  }
+                  op = op.call(tid, environment);
+                } else
+                  continue;
+              } catch (InterruptedException ie) {
+                ieSeen = true;
+                throw ie;
+              } finally {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.remove(tid);
+                }
+                // It's possible that cancel was called between
+                // op.call and the synchronized block above. Do a last
+                // check to see if that's the case.
+                if (!ieSeen && Thread.currentThread().isInterrupted()) {
+                  throw new InterruptedException("FaTE thread was interrupted");
+                }

Review comment:
       resolved in [111820f](https://github.com/apache/accumulo/pull/2467/commits/111820f8e93b0163e9bc69194a8442dfc3522e4b)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r803834774



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       This PR has been modified (2) and I have submitted issues #2482 (1) and #2483 (3)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r802851767



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       I think a path forward might be to:
   
   1. Verify that compaction cancellations are working in 1.x and 2.1.0 (new GH issue)
   2. Modify this PR to only cancel FaTE transactions in the submitted state
   3. Research range lock implementation  (new GH issue)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion merged pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion merged pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800766352



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       I'm not tied to the name, we can change it to whatever... Doesn't the currently named `fail` method actually cancel ? I was trying to stick with the current naming convention.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801090348



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,24 +83,43 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
-              deferTime = op.isReady(tid, environment);
-
-              if (deferTime == 0) {
-                prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+              try {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
                 }
-                op = op.call(tid, environment);
-              } else
-                continue;
-
+                deferTime = op.isReady(tid, environment);
+                if (deferTime == 0) {
+                  prevOp = op;
+                  if (status == SUBMITTED) {
+                    store.setStatus(tid, IN_PROGRESS);
+                  }
+                  op = op.call(tid, environment);
+                } else
+                  continue;
+              } finally {
+                synchronized (Thread.currentThread()) {
+                  synchronized (RUNNING_TRANSACTIONS) {
+                    RUNNING_TRANSACTIONS.remove(tid);
+                  }
+                }

Review comment:
       Incorporated this change into [c49f55e](https://github.com/apache/accumulo/pull/2467/commits/c49f55ec90ed3ea883c9dbfd22dfac740b3e8b3d)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#issuecomment-1031543508


   In c9c12ec I created ManagerClient.cancelFateOperation and modified FateCommand to call that. Added double-check in FateCommand that the user wants to cancel the txid. Added server side check that user can perform system actions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800898523



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,35 +83,58 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
+              synchronized (RUNNING_TRANSACTIONS) {
+                RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
+              }
               deferTime = op.isReady(tid, environment);
 
               if (deferTime == 0) {
                 prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+                if (status == SUBMITTED) {
+                  store.setStatus(tid, IN_PROGRESS);
                 }
                 op = op.call(tid, environment);
               } else
                 continue;
 
             } catch (Exception e) {
+              if (e instanceof InterruptedException) {
+                runnerLog.info(
+                    "FATE Runner thread interrupted processing txid " + Long.toHexString(tid));
+                if (prevOp != null) {
+                  processFailed(tid, prevOp);
+                }
+                if (op != prevOp) {
+                  processFailed(tid, op);
+                }

Review comment:
       Ok, so you are thinking I should change `processFailed` to `transitionToFailed` on lines 116 and 119?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800837861



##########
File path: test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
##########
@@ -71,35 +76,63 @@ public TestOperation(NamespaceId namespaceId, TableId tableId) {
 
     @Override
     public long isReady(long tid, Manager manager) throws Exception {
-      return Utils.reserveNamespace(manager, namespaceId, tid, false, true, TableOperation.RENAME)
-          + Utils.reserveTable(manager, tableId, tid, true, true, TableOperation.RENAME);
+      LOG.info("Entering isReady {}", Long.toHexString(tid));
+      try {
+        FateIT.inReady();
+        return Utils.reserveNamespace(manager, namespaceId, tid, false, true, TableOperation.RENAME)
+            + Utils.reserveTable(manager, tableId, tid, true, true, TableOperation.RENAME);
+      } finally {
+        LOG.info("Leaving isReady {}", Long.toHexString(tid));
+      }
     }
 
     @Override
     public void undo(long tid, Manager manager) throws Exception {
-      Utils.unreserveNamespace(manager, namespaceId, tid, false);
-      Utils.unreserveTable(manager, tableId, tid, true);
+      LOG.info("Entering undo {}", Long.toHexString(tid));
+      try {
+        Utils.unreserveNamespace(manager, namespaceId, tid, false);
+        Utils.unreserveTable(manager, tableId, tid, true);
+      } finally {
+        LOG.info("Leaving undo {}", Long.toHexString(tid));
+      }
     }
 
     @Override
-    public Repo<Manager> call(long tid, Manager environment) throws Exception {
-      FateIT.inCall();
-      return null;
+    public Repo<Manager> call(long tid, Manager manager) throws Exception {
+      LOG.info("Entering call {}", Long.toHexString(tid));
+      try {
+        FateIT.inCall();
+        return null;
+      } finally {
+        Utils.unreserveNamespace(manager, namespaceId, tid, false);
+        Utils.unreserveTable(manager, tableId, tid, true);
+        LOG.info("Leaving call {}", Long.toHexString(tid));

Review comment:
       Addressed in c9e64f4




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r822995334



##########
File path: core/src/main/java/org/apache/accumulo/core/clientImpl/ManagerClient.java
##########
@@ -193,4 +195,32 @@ public static void executeVoid(ClientContext context,
       throw new AssertionError(e);
     }
   }
+
+  public static boolean cancelFateOperation(ClientContext context, Long txid)
+      throws AccumuloException, AccumuloSecurityException {
+    ManagerClientService.Client client = null;
+    while (true) {
+      try {
+        client = getConnectionWithRetry(context);
+        return client.cancelFateOperation(TraceUtil.traceInfo(), context.rpcCreds(), txid);
+      } catch (TTransportException tte) {
+        log.debug("ManagerClient request failed, retrying ... ", tte);
+        sleepUninterruptibly(100, TimeUnit.MILLISECONDS);
+      } catch (ThriftSecurityException e) {
+        throw new AccumuloSecurityException(e.user, e.code, e);
+      } catch (ThriftTableOperationException e) {
+        throw new AccumuloException(e);
+      } catch (ThriftNotActiveServiceException e) {
+        // Let it loop, fetching a new location
+        log.debug("Contacted a Manager which is no longer active, re-creating"
+            + " the connection to the active Manager");
+      } catch (Exception e) {
+        throw new AccumuloException(e);
+      } finally {
+        if (client != null)
+          close(client, context);

Review comment:
       I committed your other suggestion




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r822711542



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,24 +83,51 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
+            boolean exceptionSeen = false;
             try {
-              deferTime = op.isReady(tid, environment);
-
-              if (deferTime == 0) {
-                prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+              try {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());

Review comment:
       This is OBE as I modified the code in 7b7d1b7 to only cancel new or submitted transactions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r822712781



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +335,55 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          t.interrupt();

Review comment:
       This is OBE as I modified the code in 7b7d1b7 to only cancel new or submitted transactions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r802585688



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       > might be taking the metaphor too far here
   
   I'm not convinced that it's the right one as it's [documented](https://accumulo.apache.org/docs/2.x/administration/fate). For me, surgery is manually inserting / updating / deleting rows from the metadata table.
   
   > Right now, this seems very risky to add, especially since we haven't even ironed out the kinks in the other recent changes in these areas. ExternalCompactions ITs are still flaky, and now FateIT is failing too.
   
   I"m starting to think the issue is ZK based on the information you added to in #2463 and the fact that FateIT has one test.
   
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] milleruntime commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800763753



##########
File path: test/src/main/java/org/apache/accumulo/test/fate/zookeeper/FateIT.java
##########
@@ -71,35 +76,63 @@ public TestOperation(NamespaceId namespaceId, TableId tableId) {
 
     @Override
     public long isReady(long tid, Manager manager) throws Exception {
-      return Utils.reserveNamespace(manager, namespaceId, tid, false, true, TableOperation.RENAME)
-          + Utils.reserveTable(manager, tableId, tid, true, true, TableOperation.RENAME);
+      LOG.info("Entering isReady {}", Long.toHexString(tid));
+      try {
+        FateIT.inReady();
+        return Utils.reserveNamespace(manager, namespaceId, tid, false, true, TableOperation.RENAME)
+            + Utils.reserveTable(manager, tableId, tid, true, true, TableOperation.RENAME);
+      } finally {
+        LOG.info("Leaving isReady {}", Long.toHexString(tid));
+      }
     }
 
     @Override
     public void undo(long tid, Manager manager) throws Exception {
-      Utils.unreserveNamespace(manager, namespaceId, tid, false);
-      Utils.unreserveTable(manager, tableId, tid, true);
+      LOG.info("Entering undo {}", Long.toHexString(tid));
+      try {
+        Utils.unreserveNamespace(manager, namespaceId, tid, false);
+        Utils.unreserveTable(manager, tableId, tid, true);
+      } finally {
+        LOG.info("Leaving undo {}", Long.toHexString(tid));
+      }
     }
 
     @Override
-    public Repo<Manager> call(long tid, Manager environment) throws Exception {
-      FateIT.inCall();
-      return null;
+    public Repo<Manager> call(long tid, Manager manager) throws Exception {
+      LOG.info("Entering call {}", Long.toHexString(tid));
+      try {
+        FateIT.inCall();
+        return null;
+      } finally {
+        Utils.unreserveNamespace(manager, namespaceId, tid, false);
+        Utils.unreserveTable(manager, tableId, tid, true);
+        LOG.info("Leaving call {}", Long.toHexString(tid));

Review comment:
       You have a bunch of `info` statements like this. Could probably log them at debug or remove them if you don't need them anymore.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] milleruntime commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
milleruntime commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800761995



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       Why not just call it cancel?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800901600



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,35 +83,58 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
+              synchronized (RUNNING_TRANSACTIONS) {
+                RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
+              }
               deferTime = op.isReady(tid, environment);
 
               if (deferTime == 0) {
                 prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+                if (status == SUBMITTED) {
+                  store.setStatus(tid, IN_PROGRESS);
                 }
                 op = op.call(tid, environment);
               } else
                 continue;
 
             } catch (Exception e) {
+              if (e instanceof InterruptedException) {
+                runnerLog.info(
+                    "FATE Runner thread interrupted processing txid " + Long.toHexString(tid));
+                if (prevOp != null) {
+                  processFailed(tid, prevOp);
+                }
+                if (op != prevOp) {
+                  processFailed(tid, op);
+                }

Review comment:
       I was thinking of just not calling processFailed here unless there is a compelling reason to do so.  When the code calls transitionToFailed then it will eventually call processFailed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800655813



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +328,56 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == TStatus.NEW || status == TStatus.SUBMITTED
+              || status == TStatus.IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(

Review comment:
       See https://github.com/apache/accumulo/blob/main/core/src/main/java/org/apache/accumulo/fate/Fate.java#L48




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] keith-turner commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
keith-turner commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800919259



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,35 +83,58 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
             try {
+              synchronized (RUNNING_TRANSACTIONS) {
+                RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
+              }
               deferTime = op.isReady(tid, environment);
 
               if (deferTime == 0) {
                 prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+                if (status == SUBMITTED) {
+                  store.setStatus(tid, IN_PROGRESS);
                 }
                 op = op.call(tid, environment);
               } else
                 continue;
 
             } catch (Exception e) {
+              if (e instanceof InterruptedException) {
+                runnerLog.info(
+                    "FATE Runner thread interrupted processing txid " + Long.toHexString(tid));
+                if (prevOp != null) {
+                  processFailed(tid, prevOp);
+                }
+                if (op != prevOp) {
+                  processFailed(tid, op);
+                }
+                Thread.interrupted();

Review comment:
       Ah, I thought it was interrupting the currently thread.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r822876990



##########
File path: core/src/main/java/org/apache/accumulo/core/clientImpl/ManagerClient.java
##########
@@ -193,4 +195,32 @@ public static void executeVoid(ClientContext context,
       throw new AssertionError(e);
     }
   }
+
+  public static boolean cancelFateOperation(ClientContext context, Long txid)
+      throws AccumuloException, AccumuloSecurityException {
+    ManagerClientService.Client client = null;
+    while (true) {

Review comment:
       Alternatively, put this null assignment inside the while loop to clear it from previous iterations.
   ```suggestion
       while (true) {
         ManagerClientService.Client client = null;
   ```

##########
File path: core/src/main/java/org/apache/accumulo/core/clientImpl/ManagerClient.java
##########
@@ -193,4 +195,32 @@ public static void executeVoid(ClientContext context,
       throw new AssertionError(e);
     }
   }
+
+  public static boolean cancelFateOperation(ClientContext context, Long txid)
+      throws AccumuloException, AccumuloSecurityException {
+    ManagerClientService.Client client = null;
+    while (true) {
+      try {
+        client = getConnectionWithRetry(context);
+        return client.cancelFateOperation(TraceUtil.traceInfo(), context.rpcCreds(), txid);
+      } catch (TTransportException tte) {
+        log.debug("ManagerClient request failed, retrying ... ", tte);
+        sleepUninterruptibly(100, TimeUnit.MILLISECONDS);
+      } catch (ThriftSecurityException e) {
+        throw new AccumuloSecurityException(e.user, e.code, e);
+      } catch (ThriftTableOperationException e) {
+        throw new AccumuloException(e);
+      } catch (ThriftNotActiveServiceException e) {
+        // Let it loop, fetching a new location
+        log.debug("Contacted a Manager which is no longer active, re-creating"
+            + " the connection to the active Manager");
+      } catch (Exception e) {
+        throw new AccumuloException(e);
+      } finally {
+        if (client != null)
+          close(client, context);

Review comment:
       Since client is re-assigned in each iteration of the loop, it'd be good to set it null here, in case the assignment fails, we don't try to re-close the value from the previous iteration.
   ```suggestion
           if (client != null) {
             close(client, context);
           }
           client = null;
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r822711305



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -71,24 +83,51 @@ public void run() {
         try {
           tid = store.reserve();
           TStatus status = store.getStatus(tid);
+          if (status == FAILED) {
+            runnerLog.info(
+                "FATE txid " + Long.toHexString(tid) + " likely cancelled before it started.");
+            return;
+          }
           Repo<T> op = store.top(tid);
-          if (status == TStatus.FAILED_IN_PROGRESS) {
+          if (status == FAILED_IN_PROGRESS) {
             processFailed(tid, op);
           } else {
             Repo<T> prevOp = null;
+            boolean exceptionSeen = false;
             try {
-              deferTime = op.isReady(tid, environment);
-
-              if (deferTime == 0) {
-                prevOp = op;
-                if (status == TStatus.SUBMITTED) {
-                  store.setStatus(tid, TStatus.IN_PROGRESS);
+              try {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.put(tid, Thread.currentThread());
                 }
-                op = op.call(tid, environment);
-              } else
-                continue;
-
+                deferTime = op.isReady(tid, environment);
+                if (deferTime == 0) {
+                  prevOp = op;
+                  if (status == SUBMITTED) {
+                    store.setStatus(tid, IN_PROGRESS);
+                  }
+                  op = op.call(tid, environment);
+                } else
+                  continue;
+              } catch (Exception e) {
+                exceptionSeen = true;
+                throw e;
+              } finally {
+                synchronized (RUNNING_TRANSACTIONS) {
+                  RUNNING_TRANSACTIONS.remove(tid);

Review comment:
       This is OBE as I modified the code in 7b7d1b7 to only cancel new or submitted transactions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] EdColeman commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801706468



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       Generally the two operations that would be useful to cancel would be compactions and bulk-import - both which can be long running operations, that after they have been submitted, things could have changed such that they are not longer desired - or at least not desired at that point in time.
   
   It would also be desirable that a delete operation take precedence over any other operation in progress or queued for a table.  If a compaction or a bulk import is running and then the system requests the table be deleted, the only consistency required is that the metadata table remains correct and the proper files get marked for gc'ing - anything else that is occurring is work that is going to be thrown away.  There is does not much benefit of having the compaction continue to read / write data, that as soon as it completes is going to be marked for deletion.  There may be subtleties that I'm ignoring if the table was cloned, but in practice, cloning seems to be rare,
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] EdColeman commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
EdColeman commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801738937



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       w.r.t cloning and allowing a compaction to complete was to highlight that case because there maybe issues that I don't foresee - say you have table_a and clone_a that was made from table_a - they have shared files and possibly some unique files.  If clone_a is compacted and then a delete request for clone_a occurs - is there any benefit to table_a if the compaction was allowed to complete? I don't "think so", but maybe?
   
   In the case of trying to clone table_a and while that was in progress a request to delete table_a is executed - allowing the clone to complete (which should be fast) seems it would best preserve operator intent - but if the clone was aborted, and table_a deleted, well, that might not be a complete surprise either, If you wanted to clone the table, you shouldn't have deleted it until you verified that the clone was complete - and maybe we just don't try to protect from that.
   
   Have clone wait on other operations on the source table would seem the best to allow operator to reason about the state of the clone.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801733342



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       I'm waiting to respond as I'm gathering use case information from users.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801868052



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       Feedback from user(s) is:
     1. cancelling a compaction from the compact command has never worked.
       > Is it broken in 1.x versions?
     2. Stopping the Manager is a problem because they can't load data via bulk imports, they have strict timeliness requirements. They have to stop bulk imports from getting kicked off, wait for the current ones to finish, then take down the Manager. Stopping the Manager also stops any long-running WAL recovery due to a failed TServer, table balancing, etc.
     3. Another case is issuing DeleteRows command to remove  some data that is causing problems system-wide. The DeleteRows operation may be blocked by less important Fate transactions which may be operating on a different range within the table.
     4. Accumulo's approach to locking the entire table seems heavy-handed.  There could be bulk imports and compactions running that affect one active part of a table (let's say a newer date) and we're trying to deleterows or merge on a completely different range that's inactive (older dates).
       > Maybe we should support Row or Range locks?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800936842



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +335,55 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          t.interrupt();

Review comment:
       What if I moved lines 373 to 381 in the synchronized block? Then, it should be guaranteed that if the transaction is in RUNNING_TRANSACTIONS, that the Thread running it has not moved on to something else. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] dlmarion commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
dlmarion commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r800980820



##########
File path: core/src/main/java/org/apache/accumulo/fate/Fate.java
##########
@@ -300,6 +335,55 @@ public TStatus waitForCompletion(long tid) {
     return store.waitForStatusChange(tid, FINISHED_STATES);
   }
 
+  /**
+   * Attempts to cancel a running Fate transaction
+   *
+   * @param tid
+   *          transaction id
+   * @return true if transaction transitioned to a failed state or already in a completed state,
+   *         false otherwise
+   */
+  public boolean cancel(long tid) {
+    String tidStr = Long.toHexString(tid);
+    for (int retries = 0; retries < 5; retries++) {
+      if (store.tryReserve(tid)) {
+        try {
+          TStatus status = store.getStatus(tid);
+          if (status == NEW || status == SUBMITTED || status == IN_PROGRESS) {
+            store.setProperty(tid, EXCEPTION_PROP, new TApplicationException(
+                TApplicationException.INTERNAL_ERROR, "Fate transaction cancelled by user"));
+            store.setStatus(tid, FAILED_IN_PROGRESS);
+            log.info(
+                "Updated status for Repo {} to FAILED_IN_PROGRESS because it was cancelled by user",
+                tidStr);
+            return true;
+          } else {
+            log.info("Repo {} cancelled by user but already in finished state", tidStr);
+            return true;
+          }
+        } finally {
+          store.unreserve(tid, 0);
+        }
+      } else {
+        // It's possible that the FateOp is being run by the TransactionRunner
+        Thread t = null;
+        synchronized (RUNNING_TRANSACTIONS) {
+          t = RUNNING_TRANSACTIONS.get(tid);
+        }
+        if (t != null) {
+          t.interrupt();

Review comment:
       I re-worked the synchronization in [6d22ea7](https://github.com/apache/accumulo/pull/2467/commits/6d22ea7708b786a9b9cd2aa4469e332e29e57476)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [accumulo] ctubbsii commented on a change in pull request #2467: Added mechanism to cancel FaTE operations from Shell with Manager up

Posted by GitBox <gi...@apache.org>.
ctubbsii commented on a change in pull request #2467:
URL: https://github.com/apache/accumulo/pull/2467#discussion_r801719195



##########
File path: shell/src/main/java/org/apache/accumulo/shell/commands/FateCommand.java
##########
@@ -136,7 +140,29 @@ public int execute(final String fullCommand, final CommandLine cl, final Shell s
         getZooReaderWriter(context, siteConfig, cl.getOptionValue(secretOption.getOpt()));
     ZooStore<FateCommand> zs = new ZooStore<>(path, zk);
 
-    if ("fail".equals(cmd)) {
+    if ("fail-live".equals(cmd)) {

Review comment:
       > Generally the two operations that would be useful to cancel would be compactions and bulk-import
   
   So, we currently have the ability to cancel compactions. I don't believe the new bulk import is long-running. So, any problems with long-running bulk imports should be alleviated by avoiding the old bulk import mechanism.
   
   > It would also be desirable that a delete operation take precedence over any other operation in progress or queued for a table.
   
   Agreed, but I'm not sure that would require exposing anything to users/admins. That should just be handled internally. I think recent work has moved in this direction.
   
   > There may be subtleties that I'm ignoring if the table was cloned
   
   Cloning should be fast, but it may wait on other operations, like compactions, just to avoid having to do its own redundant compactions. I'm not sure. I'd have to check the code, but a sensible API for that might be to add an option to wait for outstanding compactions (or to not wait).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org