You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@geode.apache.org by GitBox <gi...@apache.org> on 2022/01/29 00:17:04 UTC

[GitHub] [geode] ezoerner opened a new pull request #7328: GEODE-9994 Make Redis RENAME atomic

ezoerner opened a new pull request #7328:
URL: https://github.com/apache/geode/pull/7328


   Most of the work here was on getting a DUnit test to fail with the old version of the code. The test doesn't fail every time when run with the old code, but it does fail sometimes. With the new code, i.e. the fix, I haven't seen the test fail at all.
   
   
   <!-- Thank you for submitting a contribution to Apache Geode. -->
   
   <!-- In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken: 
   -->
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
   
   - [ ] Has your PR been rebased against the latest commit within the target branch (typically `develop`)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   - [ ] Does `gradlew build` run cleanly?
   
   - [ ] Have you written or updated unit tests to verify your changes?
   
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   
   <!-- Note:
   Please ensure that once the PR is submitted, check Concourse for build issues and
   submit an update to your PR as soon as possible. If you need help, please send an
   email to dev@geode.apache.org.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] jdeppe-pivotal merged pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal merged pull request #7328:
URL: https://github.com/apache/geode/pull/7328


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner edited a comment on pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner edited a comment on pull request #7328:
URL: https://github.com/apache/geode/pull/7328#issuecomment-1028165833


   I've modified the verification to allow for erroneous client reties on successful renames that appear to the client to be failures due to crashing server


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner commented on a change in pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner commented on a change in pull request #7328:
URL: https://github.com/apache/geode/pull/7328#discussion_r794969847



##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/key/RenameDUnitTest.java
##########
@@ -210,6 +227,48 @@ private void doConcurrentRenames(List<String> listOfKeys1, List<String> listOfKe
     future4.get();
   }
 
+  @Test
+  public void givenCrashDuringRename_thenDoesNotLeaveInconsistencies() throws Exception {
+    final AtomicBoolean running = new AtomicBoolean(true);
+
+    final List<String> hashtags = new ArrayList<>();
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 1));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 2));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 3));
+
+    final Runnable task1 = () -> renamePerformAndVerify(1, 10000, hashtags.get(0), running, true);
+    final Runnable task2 = () -> renamePerformAndVerify(2, 10000, hashtags.get(1), running, true);
+    final Runnable task3 = () -> renamePerformAndVerify(3, 10000, hashtags.get(2), running, true);
+
+    final Future<Void> future1 = executor.runAsync(task1);
+    final Future<Void> future2 = executor.runAsync(task2);
+    final Future<Void> future3 = executor.runAsync(task3);
+
+    final String finalRedisPort = Integer.toString(server3Port);
+    final int finalLocatorPort = locatorPort;
+    final Future<?> crasherFuture = executor.submit(() -> {
+      try {
+        for (int i = 0; i < 20 && running.get(); i++) {
+          clusterStartUp.moveBucketForKey(hashtags.get(0), "server-3");

Review comment:
       This follows the pattern in MSetDUnitTest `testMSet_crashDoesNotLeaveInconsistencies` which Jens wrote. I believe the purpose of this line to make sure there are keys hosted on the server that is about to be crashed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7328:
URL: https://github.com/apache/geode/pull/7328#discussion_r796189373



##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/key/RenameDUnitTest.java
##########
@@ -210,6 +227,48 @@ private void doConcurrentRenames(List<String> listOfKeys1, List<String> listOfKe
     future4.get();
   }
 
+  @Test
+  public void givenCrashDuringRename_thenDoesNotLeaveInconsistencies() throws Exception {
+    final AtomicBoolean running = new AtomicBoolean(true);
+
+    final List<String> hashtags = new ArrayList<>();
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 1));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 2));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 3));
+
+    final Runnable task1 = () -> renamePerformAndVerify(1, 10000, hashtags.get(0), running, true);
+    final Runnable task2 = () -> renamePerformAndVerify(2, 10000, hashtags.get(1), running, true);
+    final Runnable task3 = () -> renamePerformAndVerify(3, 10000, hashtags.get(2), running, true);
+
+    final Future<Void> future1 = executor.runAsync(task1);
+    final Future<Void> future2 = executor.runAsync(task2);
+    final Future<Void> future3 = executor.runAsync(task3);
+
+    final String finalRedisPort = Integer.toString(server3Port);
+    final int finalLocatorPort = locatorPort;
+    final Future<?> crasherFuture = executor.submit(() -> {
+      try {
+        for (int i = 0; i < 20 && running.get(); i++) {
+          clusterStartUp.moveBucketForKey(hashtags.get(0), "server-3");

Review comment:
       The point was to always ensure we're crashing the server that actually has the key. It seems easier to move the bucket (key) to a known server and then do the stop/start instead of figuring out which server is hosting the key and then perform the stop/start against a varying server.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner edited a comment on pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner edited a comment on pull request #7328:
URL: https://github.com/apache/geode/pull/7328#issuecomment-1028320000


   Just waiting on all the checks to complete, and in the meantime I am re-running the current version of the test locally with the old non-transactional code to try and make it fail. Regardless though, once the checks pass I think this can be merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner commented on a change in pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner commented on a change in pull request #7328:
URL: https://github.com/apache/geode/pull/7328#discussion_r794971759



##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/key/RenameDUnitTest.java
##########
@@ -210,6 +227,48 @@ private void doConcurrentRenames(List<String> listOfKeys1, List<String> listOfKe
     future4.get();
   }
 
+  @Test
+  public void givenCrashDuringRename_thenDoesNotLeaveInconsistencies() throws Exception {
+    final AtomicBoolean running = new AtomicBoolean(true);
+
+    final List<String> hashtags = new ArrayList<>();
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 1));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 2));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 3));
+
+    final Runnable task1 = () -> renamePerformAndVerify(1, 10000, hashtags.get(0), running, true);
+    final Runnable task2 = () -> renamePerformAndVerify(2, 10000, hashtags.get(1), running, true);
+    final Runnable task3 = () -> renamePerformAndVerify(3, 10000, hashtags.get(2), running, true);
+
+    final Future<Void> future1 = executor.runAsync(task1);
+    final Future<Void> future2 = executor.runAsync(task2);
+    final Future<Void> future3 = executor.runAsync(task3);
+
+    final String finalRedisPort = Integer.toString(server3Port);
+    final int finalLocatorPort = locatorPort;
+    final Future<?> crasherFuture = executor.submit(() -> {
+      try {
+        for (int i = 0; i < 20 && running.get(); i++) {
+          clusterStartUp.moveBucketForKey(hashtags.get(0), "server-3");

Review comment:
       Perhaps when the server has previously crashed there may not be keys hosted there anymore. Maybe Jens can comment on this as well if there's more to it than this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] DonalEvans commented on a change in pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
DonalEvans commented on a change in pull request #7328:
URL: https://github.com/apache/geode/pull/7328#discussion_r794968280



##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/key/RenameDUnitTest.java
##########
@@ -210,6 +227,48 @@ private void doConcurrentRenames(List<String> listOfKeys1, List<String> listOfKe
     future4.get();
   }
 
+  @Test
+  public void givenCrashDuringRename_thenDoesNotLeaveInconsistencies() throws Exception {
+    final AtomicBoolean running = new AtomicBoolean(true);
+
+    final List<String> hashtags = new ArrayList<>();
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 1));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 2));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 3));
+
+    final Runnable task1 = () -> renamePerformAndVerify(1, 10000, hashtags.get(0), running, true);
+    final Runnable task2 = () -> renamePerformAndVerify(2, 10000, hashtags.get(1), running, true);
+    final Runnable task3 = () -> renamePerformAndVerify(3, 10000, hashtags.get(2), running, true);
+
+    final Future<Void> future1 = executor.runAsync(task1);
+    final Future<Void> future2 = executor.runAsync(task2);
+    final Future<Void> future3 = executor.runAsync(task3);
+
+    final String finalRedisPort = Integer.toString(server3Port);
+    final int finalLocatorPort = locatorPort;
+    final Future<?> crasherFuture = executor.submit(() -> {
+      try {
+        for (int i = 0; i < 20 && running.get(); i++) {
+          clusterStartUp.moveBucketForKey(hashtags.get(0), "server-3");

Review comment:
       Is this necessary? It seems like crashing the server should be sufficient to trigger the bug, without needing to move the bucket first.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner commented on pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner commented on pull request #7328:
URL: https://github.com/apache/geode/pull/7328#issuecomment-1024773531


   Also, the error messages in the DUnit test need to be improved so they reflect the boolean logic correctly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner commented on pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner commented on pull request #7328:
URL: https://github.com/apache/geode/pull/7328#issuecomment-1024771831


   Hmm, just after I submitted this PR I ran the test again locally and it failed because the new key existed even though the RENAME failed.  I will have to take a closer look at this to make sure it's not a logic error in the test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner commented on pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner commented on pull request #7328:
URL: https://github.com/apache/geode/pull/7328#issuecomment-1024772773


   The build failure here appears to be unrelated (a groovy error?)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner commented on pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner commented on pull request #7328:
URL: https://github.com/apache/geode/pull/7328#issuecomment-1028320000


   Just waiting on all the checks to complete, and in the meantime I am re-running the current version of the test with the old non-transactional code to try and make it fail. Regardless though, once the checks pass I think this can be merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner commented on a change in pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner commented on a change in pull request #7328:
URL: https://github.com/apache/geode/pull/7328#discussion_r797837970



##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/key/RenameDUnitTest.java
##########
@@ -239,24 +298,33 @@ public void givenBucketsMoveDuringRename_thenDataIsNotLost() throws Exception {
     future3.get();
   }
 
-  private void renamePerformAndVerify(int index, int minimumIterations, String hashtag,
-      AtomicBoolean isRunning) {
-    String baseKey = "{" + hashtag + "}-key-" + index;
+  private void renamePerformAndVerify(final int index, final int minimumIterations,
+      final String hashtag, final AtomicBoolean isRunning, final boolean continueOnError) {
+    final String baseKey = "{" + hashtag + "}-key-" + index;
     jedisCluster.set(baseKey + "-0", "value");
     int iterationCount = 0;
 
     while (iterationCount < minimumIterations || isRunning.get()) {
-      String oldKey = baseKey + "-" + iterationCount;
-      String newKey = baseKey + "-" + (iterationCount + 1);
+      final String oldKey = baseKey + "-" + iterationCount;
+      final String newKey = baseKey + "-" + (iterationCount + 1);
+
+      // it's possible previous rename failed, so make sure oldKey exists
+      jedisCluster.setnx(oldKey, "value");
+
       try {
         jedisCluster.rename(oldKey, newKey);
-      } catch (Exception ex) {
-        isRunning.set(false);
-        throw new RuntimeException("Exception performing RENAME " + oldKey + " " + newKey, ex);
+      } catch (final Exception ex) {
+        if (continueOnError) {
+          logger.error("Exception performing RENAME " + oldKey + " " + newKey, ex);

Review comment:
       ah, didn't know that, I'll change that




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7328:
URL: https://github.com/apache/geode/pull/7328#discussion_r797828190



##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/key/RenameDUnitTest.java
##########
@@ -239,24 +298,33 @@ public void givenBucketsMoveDuringRename_thenDataIsNotLost() throws Exception {
     future3.get();
   }
 
-  private void renamePerformAndVerify(int index, int minimumIterations, String hashtag,
-      AtomicBoolean isRunning) {
-    String baseKey = "{" + hashtag + "}-key-" + index;
+  private void renamePerformAndVerify(final int index, final int minimumIterations,
+      final String hashtag, final AtomicBoolean isRunning, final boolean continueOnError) {
+    final String baseKey = "{" + hashtag + "}-key-" + index;
     jedisCluster.set(baseKey + "-0", "value");
     int iterationCount = 0;
 
     while (iterationCount < minimumIterations || isRunning.get()) {
-      String oldKey = baseKey + "-" + iterationCount;
-      String newKey = baseKey + "-" + (iterationCount + 1);
+      final String oldKey = baseKey + "-" + iterationCount;
+      final String newKey = baseKey + "-" + (iterationCount + 1);
+
+      // it's possible previous rename failed, so make sure oldKey exists
+      jedisCluster.setnx(oldKey, "value");
+
       try {
         jedisCluster.rename(oldKey, newKey);
-      } catch (Exception ex) {
-        isRunning.set(false);
-        throw new RuntimeException("Exception performing RENAME " + oldKey + " " + newKey, ex);
+      } catch (final Exception ex) {
+        if (continueOnError) {
+          logger.error("Exception performing RENAME " + oldKey + " " + newKey, ex);

Review comment:
       If this message is logged at the `error` level it will cause DUnit suspect processing to fail the test. Should this perhaps just be `warn`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ezoerner commented on pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
ezoerner commented on pull request #7328:
URL: https://github.com/apache/geode/pull/7328#issuecomment-1028165833


   I've modified the verification to allow for erroneous client reties on successful renames that appear to be errors


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] jdeppe-pivotal commented on a change in pull request #7328: GEODE-9994 Make Redis RENAME atomic

Posted by GitBox <gi...@apache.org>.
jdeppe-pivotal commented on a change in pull request #7328:
URL: https://github.com/apache/geode/pull/7328#discussion_r796189373



##########
File path: geode-for-redis/src/distributedTest/java/org/apache/geode/redis/internal/commands/executor/key/RenameDUnitTest.java
##########
@@ -210,6 +227,48 @@ private void doConcurrentRenames(List<String> listOfKeys1, List<String> listOfKe
     future4.get();
   }
 
+  @Test
+  public void givenCrashDuringRename_thenDoesNotLeaveInconsistencies() throws Exception {
+    final AtomicBoolean running = new AtomicBoolean(true);
+
+    final List<String> hashtags = new ArrayList<>();
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 1));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 2));
+    hashtags.add(clusterStartUp.getKeyOnServer("rename", 3));
+
+    final Runnable task1 = () -> renamePerformAndVerify(1, 10000, hashtags.get(0), running, true);
+    final Runnable task2 = () -> renamePerformAndVerify(2, 10000, hashtags.get(1), running, true);
+    final Runnable task3 = () -> renamePerformAndVerify(3, 10000, hashtags.get(2), running, true);
+
+    final Future<Void> future1 = executor.runAsync(task1);
+    final Future<Void> future2 = executor.runAsync(task2);
+    final Future<Void> future3 = executor.runAsync(task3);
+
+    final String finalRedisPort = Integer.toString(server3Port);
+    final int finalLocatorPort = locatorPort;
+    final Future<?> crasherFuture = executor.submit(() -> {
+      try {
+        for (int i = 0; i < 20 && running.get(); i++) {
+          clusterStartUp.moveBucketForKey(hashtags.get(0), "server-3");

Review comment:
       The point was to always ensure we're crashing the server that actually has the key. It seems easier to move the bucket to a known server and then do the stop/start instead of figuring out which server is hosting the key and then perform the stop/start against a varying server.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@geode.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org