You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@geode.apache.org by GitBox <gi...@apache.org> on 2021/05/17 15:40:50 UTC

[GitHub] [geode] DonalEvans opened a new pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

DonalEvans opened a new pull request #6487:
URL: https://github.com/apache/geode/pull/6487


    - Replace uses of ByteArrayWrapper with byte[] in RedisSet,
    NullRedisSet and other Set-related classes
    - Replace internal Set in RedisSet with fastutil
    ObjectOpenCustomHashSet
    - Replace the membersAddAll and membersRemoveAll methods with looped
    calls to membersAdd and membersRemove, as ObjectOpenCustomHashSet does
    not implement addAll and removeAll, which causes errors when comparing
    the contents of Sets
    - Replace uses of implementations with interfaces where possible
    - These changes break the sizing tests, but these tests have been
    determined to be inconsistent due to the current implementation of
    sizeable in RedisSet. GEODE-9279 will address the issues with sizeable
    in a future commit
   
   Authored-by: Donal Evans <do...@vmware.com>
   
   Thank you for submitting a contribution to Apache Geode.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
   
   - [x] Has your PR been rebased against the latest commit within the target branch (typically `develop`)?
   
   - [x] Is your initial contribution a single, squashed commit?
   
   - [x] Does `gradlew build` run cleanly?
   
   - [x] Have you written or updated unit tests to verify your changes?
   
   - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   
   ### Note:
   Please ensure that once the PR is submitted, check Concourse for build issues and
   submit an update to your PR as soon as possible. If you need help, please send an
   email to dev@geode.apache.org.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] sabbey37 commented on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
sabbey37 commented on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-843372066


   > I don't know too much about the TCL test, but I thought this was a refactoring that shouldn't have affected them. What are the changes to the TCL tests about?
   
   The Native Redis test for `EXPIRE - write on expire should work` were utilizing `lpush`.  We did not have that command/the list data structure implemented, so at some point they were changed to do a `sadd`.  The tests also checks the result of `smembers`.  Since sets don't have a fixed order, it failed (but seems like this happened very infrequently since it was only just caught).  The test was changed to use a `set` on a string (it didn't seem to matter which data set was used, since the test just verified that we could continue to edit the value after an expiration was set).  That was all that was changed in the tcl tests, but the whole patch file was replaced (including all the same changes as before) vs. trying to edit it directly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] ringles commented on a change in pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
ringles commented on a change in pull request #6487:
URL: https://github.com/apache/geode/pull/6487#discussion_r634520633



##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/NullRedisSet.java
##########
@@ -122,30 +118,38 @@ private int doSetOpWhileLocked(SetOp setOp, CommandHelper helper,
     }
   }
 
-  private Set<ByteArrayWrapper> computeSetOp(SetOp setOp,
-      ArrayList<Set<ByteArrayWrapper>> nonDestinationSets,
-      CommandHelper helper,
-      RedisKey destination) {
-    Set<ByteArrayWrapper> result = null;
+  private Set<byte[]> computeSetOp(SetOp setOp, List<Set<byte[]>> nonDestinationSets,
+      CommandHelper helper, RedisKey destination) {
+    ObjectOpenCustomHashSet<byte[]> result = null;
     if (nonDestinationSets.isEmpty()) {
       return emptySet();
     }
-    for (Set<ByteArrayWrapper> set : nonDestinationSets) {
+    for (Set<byte[]> set : nonDestinationSets) {
       if (set == null) {
         set = helper.getRedisSet(destination, false).smembers();
       }
       if (result == null) {
-        result = set;
+        result = new ObjectOpenCustomHashSet<>(set, ByteArrays.HASH_STRATEGY);
       } else {
         switch (setOp) {
           case UNION:
             result.addAll(set);
             break;
           case INTERSECTION:
-            result.retainAll(set);
+            List<byte[]> membersToRemove = new ArrayList<>();

Review comment:
       You might pull this and DIFF into sub-methods, just to make it a little easier to read. Not critical, but nice-to-have.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] lgtm-com[bot] commented on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-842471580


   This pull request **introduces 2 alerts** when merging ed22209cbd36ba25eaa9a87fe160e69fd39199df into e52217ff06f5d1643ede4c3494718fc8bab80f33 - [view on LGTM.com](https://lgtm.com/projects/g/apache/geode/rev/pr-2ae475228c62757a91e911a2c8d5c43add7db1fe)
   
   **new alerts:**
   
   * 2 for Cast from abstract to concrete collection


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] DonalEvans commented on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
DonalEvans commented on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-847351924


   > Looks solid. I think that @upthewaterspout's comments are valuable, but given the scope of this ticket, I think it would be better to add something to Jira to track that work.
   
   Agreed. I'll create a ticket.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] upthewaterspout commented on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
upthewaterspout commented on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-843368462


   I don't know too much about the TCL test, but I thought this was a refactoring that shouldn't have affected them. What are the changes to the TCL tests about?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] DonalEvans commented on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
DonalEvans commented on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-847262731


   Regarding the set copying that Dan mentioned, I agree that it's something we want to avoid and get rid of, but the sets used in the diff and intersection operations need to be a class that can handle comparisons between byte arrays in the way we want. However, returning a `ObjectOpenCustomHashSet` from the lambdas in `RedisSetCommandsFunctionExecutor.internalsmembers()` or `RedisSetCommandsFunctionExecutor.smembers()` results in serialization filter errors, so as a compromise, this set copying behaviour is used to allow a `HashSet` to be returned from the lambda and then turned back into a `ObjectOpenCustomHashSet` when needed for diff and intersection. Since these commands aren't currently supported, this bevaiour should be revisited when they're fully implemented and supported.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] DonalEvans merged pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
DonalEvans merged pull request #6487:
URL: https://github.com/apache/geode/pull/6487


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] upthewaterspout commented on a change in pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
upthewaterspout commented on a change in pull request #6487:
URL: https://github.com/apache/geode/pull/6487#discussion_r634687946



##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/NullRedisSet.java
##########
@@ -62,21 +65,19 @@ public int scard() {
   }
 
   @Override
-  long sadd(ArrayList<ByteArrayWrapper> membersToAdd, Region<RedisKey, RedisData> region,
-      RedisKey key) {
+  long sadd(List<byte[]> membersToAdd, Region<RedisKey, RedisData> region, RedisKey key) {
     region.create(key, new RedisSet(membersToAdd));
     return membersToAdd.size();
   }
 
   @Override
-  long srem(ArrayList<ByteArrayWrapper> membersToRemove, Region<RedisKey, RedisData> region,
-      RedisKey key) {
+  long srem(List<byte[]> membersToRemove, Region<RedisKey, RedisData> region, RedisKey key) {
     return 0;
   }
 
   @Override
   @VisibleForTesting
-  Set<ByteArrayWrapper> smembers() {
+  public Set<byte[]> smembers() {
     // some callers want to be able to modify the set returned
     return new HashSet<>();

Review comment:
       Should this be an Object2Object... set instead of a HashMap<>? If the callers modify this returned set, they won't be getting the right hashing behavior.

##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -340,7 +306,7 @@ long srem(ArrayList<ByteArrayWrapper> membersToRemove, Region<RedisKey, RedisDat
    * @return a set containing all the members in this set
    */
   @VisibleForTesting
-  Set<ByteArrayWrapper> smembers() {
+  public Set<byte[]> smembers() {
     return new HashSet<>(members);

Review comment:
       This definitely needs to be an Object2ObjectOpenCustomHashMap, not a HashMap here.

##########
File path: geode-apis-compatible-with-redis/src/test/java/org/apache/geode/redis/internal/data/RedisSetTest.java
##########
@@ -193,9 +197,10 @@ public void should_calculateSize_equalToROS_withNoMembers() {
   }
 
   @Test
+  @Ignore("Sizing tests are known to be flaky/incorrect and will be fixed as part of GEODE-9279")
   public void should_calculateSize_equalToROS_withSingleMember() {
-    HashSet<ByteArrayWrapper> members = new HashSet<>();
-    members.add(new ByteArrayWrapper("value".getBytes()));
+    Set<byte[]> members = new HashSet<>();

Review comment:
       I guess these tests are technically correct, but it still seems weird to use a HashSet<byte[]>() here. That will use the Object.hashCode() of the byte[] array. Maybe make this a ArrayList<> instead to make it clear we don't really care about set semantics on `members` here? Or use Object2ObjectOpenCustomHashMap instead.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] sabbey37 edited a comment on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
sabbey37 edited a comment on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-843372066


   > I don't know too much about the TCL test, but I thought this was a refactoring that shouldn't have affected them. What are the changes to the TCL tests about?
   
   The Native Redis test for `EXPIRE - write on expire should work` was utilizing `lpush`.  We did not have that command/the list data structure implemented, so at some point it was changed to do a `sadd`.  The test also checks the result of `smembers`.  Since sets don't have a fixed order, it failed (but seems like this happened very infrequently since it was only just caught).  The test was changed to use a `set` on a string (it didn't seem to matter which data structure was used, since the test just verified that we could continue to edit the value after an expiration was set).  That was all that was changed in the tcl tests, but the whole patch file was replaced (including all the same changes as before) vs. trying to edit it directly.
   
   Looks like some set Native Redis tests are failing with the new changes though.  I'm not sure what that's about (haven't looked at the full PR yet), but shouldn't have anything to do with the patch file change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] DonalEvans commented on a change in pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
DonalEvans commented on a change in pull request #6487:
URL: https://github.com/apache/geode/pull/6487#discussion_r634808156



##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -340,7 +306,7 @@ long srem(ArrayList<ByteArrayWrapper> membersToRemove, Region<RedisKey, RedisDat
    * @return a set containing all the members in this set
    */
   @VisibleForTesting
-  Set<ByteArrayWrapper> smembers() {
+  public Set<byte[]> smembers() {
     return new HashSet<>(members);

Review comment:
       The commit prior to the current head returned an `ObjectOpenCustomHashSet` here (and in NullRedisSet) but this caused serialization issues in DUnit tests as that class wasn't allowed by the serialization filter. After trying to find an alternate solution, I've settled on extending `ObjectOpenCustomHashSet` to allow adding it to the appropriate sanctioned-serializables file, and to allow proper support of `addAll()` and `removeAll()`, since the current implementation of `ObjectOpenCustomHashSet` doesn't employ the hashing strategy set in the constructor for those.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] sabbey37 edited a comment on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
sabbey37 edited a comment on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-843372066


   > I don't know too much about the TCL test, but I thought this was a refactoring that shouldn't have affected them. What are the changes to the TCL tests about?
   
   The Native Redis test for `EXPIRE - write on expire should work` were utilizing `lpush`.  We did not have that command/the list data structure implemented, so at some point they were changed to do a `sadd`.  The tests also checks the result of `smembers`.  Since sets don't have a fixed order, it failed (but seems like this happened very infrequently since it was only just caught).  The test was changed to use a `set` on a string (it didn't seem to matter which data set was used, since the test just verified that we could continue to edit the value after an expiration was set).  That was all that was changed in the tcl tests, but the whole patch file was replaced (including all the same changes as before) vs. trying to edit it directly.
   
   Looks like some set Native Redis tests are failing with the new changes though.  I'm not sure what that's about (haven't looked at the full PR yet), but shouldn't have anything to do with the patch file change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] sabbey37 edited a comment on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
sabbey37 edited a comment on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-843372066


   > I don't know too much about the TCL test, but I thought this was a refactoring that shouldn't have affected them. What are the changes to the TCL tests about?
   
   The Native Redis test for `EXPIRE - write on expire should work` were utilizing `lpush`.  We did not have that command/the list data structure implemented, so at some point they were changed to do a `sadd`.  The tests also checks the result of `smembers`.  Since sets don't have a fixed order, it failed (but seems like this happened very infrequently since it was only just caught).  The test was changed to use a `set` on a string (it didn't seem to matter which data structure was used, since the test just verified that we could continue to edit the value after an expiration was set).  That was all that was changed in the tcl tests, but the whole patch file was replaced (including all the same changes as before) vs. trying to edit it directly.
   
   Looks like some set Native Redis tests are failing with the new changes though.  I'm not sure what that's about (haven't looked at the full PR yet), but shouldn't have anything to do with the patch file change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] upthewaterspout commented on a change in pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
upthewaterspout commented on a change in pull request #6487:
URL: https://github.com/apache/geode/pull/6487#discussion_r634714167



##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -340,7 +306,7 @@ long srem(ArrayList<ByteArrayWrapper> membersToRemove, Region<RedisKey, RedisDat
    * @return a set containing all the members in this set
    */
   @VisibleForTesting
-  Set<ByteArrayWrapper> smembers() {
+  public Set<byte[]> smembers() {
     return new HashSet<>(members);

Review comment:
       I would actually suggest changing this to `Collections.unmodifiableSet(members)` here. This is an expensive operation, so it would be better if we only make a copy of this set where we actually need to modify it. Or only currently supported command, SMEMBERS, doesn't need this extra copying.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] DonalEvans commented on a change in pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
DonalEvans commented on a change in pull request #6487:
URL: https://github.com/apache/geode/pull/6487#discussion_r634816154



##########
File path: geode-apis-compatible-with-redis/src/main/java/org/apache/geode/redis/internal/data/RedisSet.java
##########
@@ -340,7 +306,7 @@ long srem(ArrayList<ByteArrayWrapper> membersToRemove, Region<RedisKey, RedisDat
    * @return a set containing all the members in this set
    */
   @VisibleForTesting
-  Set<ByteArrayWrapper> smembers() {
+  public Set<byte[]> smembers() {
     return new HashSet<>(members);

Review comment:
       Correction, the hashing strategy is correctly employed in the current implementation, it's just the serialization issue that's present. Nevertheless, extending the class allows serialization and accurate calculation of size down the line when we revisit sizeable for RedisSet, so it's still the best choice, I think.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] DonalEvans commented on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
DonalEvans commented on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-849832834


   Force pushed to rebase on develop, will merge once pre-checkin is finished.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] lgtm-com[bot] commented on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-842471580


   This pull request **introduces 2 alerts** when merging ed22209cbd36ba25eaa9a87fe160e69fd39199df into e52217ff06f5d1643ede4c3494718fc8bab80f33 - [view on LGTM.com](https://lgtm.com/projects/g/apache/geode/rev/pr-2ae475228c62757a91e911a2c8d5c43add7db1fe)
   
   **new alerts:**
   
   * 2 for Cast from abstract to concrete collection


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [geode] upthewaterspout commented on pull request #6487: GEODE-9222: Remove ByteArrayWrapper from RedisSet

Posted by GitBox <gi...@apache.org>.
upthewaterspout commented on pull request #6487:
URL: https://github.com/apache/geode/pull/6487#issuecomment-843495399


   > > I don't know too much about the TCL test, but I thought this was a refactoring that shouldn't have affected them. What are the changes to the TCL tests about?
   > 
   > The Native Redis test for `EXPIRE - write on expire should work` was utilizing `lpush`. We did not have that command/the list data structure implemented, so at some point it was changed to do a `sadd`. The test also checks the result of `smembers`. Since sets don't have a fixed order, it failed (but seems like this happened very infrequently since it was only just caught). The test was changed to use a `set` on a string (it didn't seem to matter which data structure was used, since the test just verified that we could continue to edit the value after an expiration was set). That was all that was changed in the tcl tests, but the whole patch file was replaced (including all the same changes as before) vs. trying to edit it directly.
   
   Ok, makes sense.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org