You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "szetszwo (via GitHub)" <gi...@apache.org> on 2024/04/23 23:06:20 UTC

[PR] HDDS-10745. Do not use BitSet for OzoneAcl.aclBitSet. [ozone]

szetszwo opened a new pull request, #6581:
URL: https://github.com/apache/ozone/pull/6581

   ## What changes were proposed in this pull request?
   
   The BitSet class used by aclBitSet is inefficient. We have to copy it in order to make aclBitSet immutable. BitSet is also inefficient in memory usage.
   
   ## What is the link to the Apache JIRA
   
   HDDS-10745
   
   ## How was this patch tested?
   
   Existing unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10745. Do not use BitSet for OzoneAcl.aclBitSet. [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #6581:
URL: https://github.com/apache/ozone/pull/6581


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10745. Do not use BitSet for OzoneAcl.aclBitSet. [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6581:
URL: https://github.com/apache/ozone/pull/6581#discussion_r1578854868


##########
hadoop-ozone/common/src/main/java/com/google/protobuf/Proto2Utils.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.google.protobuf;
+
+/** Utilities for protobuf v2. */
+public final class Proto2Utils {
+  /**
+   * Similar to {@link ByteString#copyFrom(byte[])} except that this method does not copy.
+   * This method is safe only if the content of the array remains unchanged.
+   * Otherwise, it violates the immutability of {@link ByteString}.
+   */
+  public static ByteString unsafeByteString(byte[] array) {
+    return new LiteralByteString(array);
+  }
+
+  private Proto2Utils() { }
+

Review Comment:
   `}` is missing from EOF.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10745. Do not use BitSet for OzoneAcl.aclBitSet. [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #6581:
URL: https://github.com/apache/ozone/pull/6581#issuecomment-2077553931

   @adoroszlai , thanks for reviewing this!  That's a good suggestion.  Let move it to `hadoop-hdds/common`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10745. Do not use BitSet for OzoneAcl.aclBitSet. [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6581:
URL: https://github.com/apache/ozone/pull/6581#discussion_r1577363794


##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/IAccessAuthorizer.java:
##########
@@ -114,6 +121,14 @@ public static ACLType getACLRight(String type) {
 
     }
 
+    public static BitSet getACLBitSet(byte[] acls) {
+      return BitSet.valueOf(acls);
+    }
+
+    public static String getACLString(byte[] acls) {
+      return getACLString(BitSet.valueOf(acls));
+    }

Review Comment:
   I suggest moving these to `OzoneAcl` and making them private.  This would make it easier to completely get rid of `BitSet` in the future by reducing the public interface.



##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OzoneAcl.java:
##########
@@ -243,47 +262,60 @@ public String getName() {
 
   @JsonIgnore
   public boolean isEmpty() {
-    return aclBitSet.isEmpty();
+    return aclBits == 0;
   }
 
   @VisibleForTesting
   public boolean isSet(ACLType acl) {
-    return aclBitSet.get(acl.ordinal());
+    return (aclBits & toInt(acl)) != 0;
   }
 
   public boolean checkAccess(ACLType acl) {
     return (isSet(acl) || isSet(ALL)) && !isSet(NONE);
   }
 
   public OzoneAcl add(OzoneAcl other) {
-    return apply(bits -> bits.or(other.aclBitSet));
+    return apply(bits -> bits | other.aclBits);
   }
 
   public OzoneAcl remove(OzoneAcl other) {
-    return apply(bits -> bits.andNot(other.aclBitSet));
+    return apply(bits -> bits & ~other.aclBits);
   }
 
-  /** @return copy of this {@code OzoneAcl} after applying the given {@code op},
-   * or this instance if {@code op} makes no difference */
-  private OzoneAcl apply(Consumer<BitSet> op) {
-    final BitSet cloneBits = copyBitSet(aclBitSet);
-    op.accept(cloneBits);
-    return cloneBits.equals(aclBitSet)
+  private OzoneAcl apply(IntFunction<Integer> op) {
+    int applied = op.apply(aclBits);
+    return applied == aclBits
         ? this
-        : new OzoneAcl(type, name, aclScope, cloneBits);
+        : new OzoneAcl(type, name, aclScope, applied);
   }
 
   @JsonIgnore
   public byte[] getAclByteArray() {
-    return aclBitSet.toByteArray();
+    // only first 9 bits are used currently
+    final byte first = (byte) aclBits;
+    final byte second = (byte) (aclBits >>> 8);
+    return second != 0 ? new byte[]{first, second} : new byte[]{first};
+  }
+
+  public List<String> getAclStringList() {
+    return getAclList(aclBits, ACLType::name);

Review Comment:
   To fix [test failure](https://github.com/apache/ozone/actions/runs/8808557708/job/24178180985?pr=6581#step:5:354):
   
   ```suggestion
     @JsonIgnore
     public List<String> getAclStringList() {
       return getAclList(aclBits, ACLType::name);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10745. Do not use BitSet for OzoneAcl.aclBitSet. [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6581:
URL: https://github.com/apache/ozone/pull/6581#issuecomment-2077708549

   Looks like failure in `TestOzoneDebugShell` is related:
   
   ```
   2024-04-25 15:58:50,480 [DBScanner-0] ERROR debug.DBScanner (DBScanner.java:call(516)) - Exception parse Object
   com.fasterxml.jackson.databind.JsonMappingException: Document nesting depth (1001) exceeds the maximum allowed (1000, from `StreamWriteConstraints.getMaxNestingDepth()`) (through reference chain: java.util.ArrayList[0]->org.apache.hadoop.ozone.OzoneAcl["toStringMethod"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.hadoop.ozone.OzoneAcl$$Lambda$832/0x00007f47105fdd68["arg$1"]->org.apache.hadoop.ozone.OzoneAcl["toStringMethod"]-...)
   	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402)
   	at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:361)
   	at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:323)
   	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:778)
   	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
   ```
   
   I think we need `@JsonIgnore` for the two new supplier fields:
   
   https://github.com/apache/ozone/blob/37cba819d7e5160919725ac0176d565134521d7b/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OzoneAcl.java#L67-L68


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org