You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "DomGarguilo (via GitHub)" <gi...@apache.org> on 2023/11/29 16:51:46 UTC

[PR] Add retry and logging to CloseScannerIT [accumulo]

DomGarguilo opened a new pull request, #3996:
URL: https://github.com/apache/accumulo/pull/3996

   This PR adds a retry to the condition we are testing for once all scanners are closed. I also added logs when the condition does not pass.
   
   I am not sure why the condition in the test is checking that the number of scans is less than 3. If anyone has insight on this maybe we can create a variable name for this number or a comment. I tried changing the condition to check that there are 0 scans and still could not get it to fail locally.
   
   This IT has failed twice recently:
   
   - https://ci-builds.apache.org/job/Accumulo/job/main/org.apache.accumulo$accumulo-test/611/testReport/junit/org.apache.accumulo.test/CloseScannerIT/testManyScans/
   - https://ci-builds.apache.org/job/Accumulo/job/main/org.apache.accumulo$accumulo-test/605/testReport/junit/org.apache.accumulo.test/CloseScannerIT/testManyScans/
   
   ```
   Error Message
   expected: <true> but was: <false>
   Stacktrace
   org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
   	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
   	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
   	at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
   	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
   	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
   	at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
   	at org.apache.accumulo.test.CloseScannerIT.testManyScans(CloseScannerIT.java:70)
   	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
   	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
   	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
   	at org.junit.jupiter.api.AssertTimeoutPreemptively.lambda$submitTask$3(AssertTimeoutPreemptively.java:95)
   	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   	at java.base/java.lang.Thread.run(Thread.java:829)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add retry and logging to CloseScannerIT [accumulo]

Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on code in PR #3996:
URL: https://github.com/apache/accumulo/pull/3996#discussion_r1413930194


##########
test/src/main/java/org/apache/accumulo/test/CloseScannerIT.java:
##########
@@ -61,14 +68,26 @@ public void testManyScans() throws Exception {
         } // when the scanner is closed, all open sessions should be closed
       }
 
-      List<String> tservers = client.instanceOperations().getTabletServers();
-      int activeScans = 0;
-      for (String tserver : tservers) {
-        activeScans += client.instanceOperations().getActiveScans(tserver).size();
-      }
+      Wait.waitFor(() -> {
+        final List<ActiveScan> activeScans = getAllActiveScans(client);
+        if (activeScans.size() < 3) {
+          return true;
+        } else {
+          LOG.error("Saw more scans than expected. Retrying. Scans found: {}", activeScans);
+          return false;
+        }
+      }, 5000, 250, "Found too many active scans after closing all scanners.");
+    }
+  }
 
-      assertTrue(activeScans < 3);
+  private static List<ActiveScan> getAllActiveScans(AccumuloClient client)

Review Comment:
   I was returning the scans so that they could be logged in the error message when the wait condition is not met. I can get rid of that and return just the count if you think that would be better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add retry and logging to CloseScannerIT [accumulo]

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #3996:
URL: https://github.com/apache/accumulo/pull/3996#discussion_r1413941402


##########
test/src/main/java/org/apache/accumulo/test/CloseScannerIT.java:
##########
@@ -61,14 +68,26 @@ public void testManyScans() throws Exception {
         } // when the scanner is closed, all open sessions should be closed
       }
 
-      List<String> tservers = client.instanceOperations().getTabletServers();
-      int activeScans = 0;
-      for (String tserver : tservers) {
-        activeScans += client.instanceOperations().getActiveScans(tserver).size();
-      }
+      Wait.waitFor(() -> {
+        final List<ActiveScan> activeScans = getAllActiveScans(client);
+        if (activeScans.size() < 3) {

Review Comment:
   Well my point was that it's not really an error so I don't think it needs to be logged in that level of detail, it's expected since it's closed async so a count seems fine at the debug level since it's expected unless someone else disagrees



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add retry and logging to CloseScannerIT [accumulo]

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #3996:
URL: https://github.com/apache/accumulo/pull/3996#discussion_r1412839102


##########
test/src/main/java/org/apache/accumulo/test/CloseScannerIT.java:
##########
@@ -61,14 +68,26 @@ public void testManyScans() throws Exception {
         } // when the scanner is closed, all open sessions should be closed
       }
 
-      List<String> tservers = client.instanceOperations().getTabletServers();
-      int activeScans = 0;
-      for (String tserver : tservers) {
-        activeScans += client.instanceOperations().getActiveScans(tserver).size();
-      }
+      Wait.waitFor(() -> {
+        final List<ActiveScan> activeScans = getAllActiveScans(client);
+        if (activeScans.size() < 3) {
+          return true;
+        } else {
+          LOG.error("Saw more scans than expected. Retrying. Scans found: {}", activeScans);
+          return false;
+        }
+      }, 5000, 250, "Found too many active scans after closing all scanners.");
+    }
+  }
 
-      assertTrue(activeScans < 3);
+  private static List<ActiveScan> getAllActiveScans(AccumuloClient client)

Review Comment:
   You can probably just return an integer here and make this method just return the total size. There doesn't seem to be much reason to return the actual scans as you just need the count. 



##########
test/src/main/java/org/apache/accumulo/test/CloseScannerIT.java:
##########
@@ -61,14 +68,26 @@ public void testManyScans() throws Exception {
         } // when the scanner is closed, all open sessions should be closed
       }
 
-      List<String> tservers = client.instanceOperations().getTabletServers();
-      int activeScans = 0;
-      for (String tserver : tservers) {
-        activeScans += client.instanceOperations().getActiveScans(tserver).size();
-      }
+      Wait.waitFor(() -> {
+        final List<ActiveScan> activeScans = getAllActiveScans(client);
+        if (activeScans.size() < 3) {
+          return true;
+        } else {
+          LOG.error("Saw more scans than expected. Retrying. Scans found: {}", activeScans);

Review Comment:
   This should likely just be a debug message or info because this is expected as the scanners are closed async. If the helper method is updated to just return the count (as I mention in my other comment) then you can just log the count here as the actual scans found don't really matter.



##########
test/src/main/java/org/apache/accumulo/test/CloseScannerIT.java:
##########
@@ -61,14 +68,26 @@ public void testManyScans() throws Exception {
         } // when the scanner is closed, all open sessions should be closed
       }
 
-      List<String> tservers = client.instanceOperations().getTabletServers();
-      int activeScans = 0;
-      for (String tserver : tservers) {
-        activeScans += client.instanceOperations().getActiveScans(tserver).size();
-      }
+      Wait.waitFor(() -> {
+        final List<ActiveScan> activeScans = getAllActiveScans(client);
+        if (activeScans.size() < 3) {

Review Comment:
   I would change this to check for 0 as with the wait implemented and retry we should always eventually be at 0.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add retry and logging to CloseScannerIT [accumulo]

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #3996:
URL: https://github.com/apache/accumulo/pull/3996#discussion_r1413941402


##########
test/src/main/java/org/apache/accumulo/test/CloseScannerIT.java:
##########
@@ -61,14 +68,26 @@ public void testManyScans() throws Exception {
         } // when the scanner is closed, all open sessions should be closed
       }
 
-      List<String> tservers = client.instanceOperations().getTabletServers();
-      int activeScans = 0;
-      for (String tserver : tservers) {
-        activeScans += client.instanceOperations().getActiveScans(tserver).size();
-      }
+      Wait.waitFor(() -> {
+        final List<ActiveScan> activeScans = getAllActiveScans(client);
+        if (activeScans.size() < 3) {

Review Comment:
   Well my point was that it's not really an error so I don't think it needs to be logged in that level of detail, it's expected since it's closed async so a count seems fine at the debug level since it's expected unless someone else disagrees



##########
test/src/main/java/org/apache/accumulo/test/CloseScannerIT.java:
##########
@@ -61,14 +68,26 @@ public void testManyScans() throws Exception {
         } // when the scanner is closed, all open sessions should be closed
       }
 
-      List<String> tservers = client.instanceOperations().getTabletServers();
-      int activeScans = 0;
-      for (String tserver : tservers) {
-        activeScans += client.instanceOperations().getActiveScans(tserver).size();
-      }
+      Wait.waitFor(() -> {
+        final List<ActiveScan> activeScans = getAllActiveScans(client);
+        if (activeScans.size() < 3) {
+          return true;
+        } else {
+          LOG.error("Saw more scans than expected. Retrying. Scans found: {}", activeScans);
+          return false;
+        }
+      }, 5000, 250, "Found too many active scans after closing all scanners.");
+    }
+  }
 
-      assertTrue(activeScans < 3);
+  private static List<ActiveScan> getAllActiveScans(AccumuloClient client)

Review Comment:
   Well my point was that it's not really an error so I don't think it needs to be logged in that level of detail, it's expected since it's closed async so a count seems fine at the debug level since it's expected unless someone else disagrees



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add retry and logging to CloseScannerIT [accumulo]

Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo commented on code in PR #3996:
URL: https://github.com/apache/accumulo/pull/3996#discussion_r1413966659


##########
test/src/main/java/org/apache/accumulo/test/CloseScannerIT.java:
##########
@@ -61,14 +68,26 @@ public void testManyScans() throws Exception {
         } // when the scanner is closed, all open sessions should be closed
       }
 
-      List<String> tservers = client.instanceOperations().getTabletServers();
-      int activeScans = 0;
-      for (String tserver : tservers) {
-        activeScans += client.instanceOperations().getActiveScans(tserver).size();
-      }
+      Wait.waitFor(() -> {
+        final List<ActiveScan> activeScans = getAllActiveScans(client);
+        if (activeScans.size() < 3) {
+          return true;
+        } else {
+          LOG.error("Saw more scans than expected. Retrying. Scans found: {}", activeScans);
+          return false;
+        }
+      }, 5000, 250, "Found too many active scans after closing all scanners.");
+    }
+  }
 
-      assertTrue(activeScans < 3);
+  private static List<ActiveScan> getAllActiveScans(AccumuloClient client)

Review Comment:
   Gotcha. Makes sense. I addressed your comments in 7a87d5b



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add retry and logging to CloseScannerIT [accumulo]

Posted by "DomGarguilo (via GitHub)" <gi...@apache.org>.
DomGarguilo merged PR #3996:
URL: https://github.com/apache/accumulo/pull/3996


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org