You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "xichen01 (via GitHub)" <gi...@apache.org> on 2024/05/08 11:35:38 UTC

[PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

xichen01 opened a new pull request, #6656:
URL: https://github.com/apache/ozone/pull/6656

   ## What changes were proposed in this pull request?
   Reduce `com/google/protobuf/ByteString.copyFromUtf8` when encode a protobuf Object.
   - For a String type protobuf field, if you use the `setXXX` method (such as `builder.setHostName`), the protobuf will call `ByteString.copyFromUtf8` convert UTF8 String to `ByteString`. 
   - For some UTF8 String with fixed content, we can cache its `ByteString` value, and use `setXXXBytes` to avoid calling `ByteString.copyFromUtf8`
   
   Before
   https://issues.apache.org/jira/secure/attachment/13068731/om-create-key-alloc-profiler-setxxxString.html
   
   After
   The `com/google/protobuf/ByteString.copyFromUtf8` in the `DatanodeDetail` disappear.
   https://issues.apache.org/jira/secure/attachment/13068732/om-create-key-alloc-profiler-setxxbytes.html
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-10811
   
   ## How was this patch tested?
   Existing Test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo merged PR #6656:
URL: https://github.com/apache/ozone/pull/6656


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6656:
URL: https://github.com/apache/ozone/pull/6656#issuecomment-2102543957

   > @xichen01 , thanks a lot for working on this! How about add a `StringWithByteString` class as below? Than, it won't have any additional cache lookups.
   > 
   > ```java
   > diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java
   > index 26f6be44e5..d62af61d2a 100644
   > --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java
   > +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java
   > @@ -801,7 +801,7 @@ public static Map<String, String> processForLogging(OzoneConfiguration conf) {
   >    }
   >  
   >    @Nonnull
   > -  public static String threadNamePrefix(@Nullable String id) {
   > +  public static String threadNamePrefix(@Nullable Object id) {
   >      return id != null && !"".equals(id)
   >          ? id + "-"
   >          : "";
   > diff --git a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
   > index b455daba52..b00c3d623d 100644
   > --- a/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
   > +++ b/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java
   > @@ -26,6 +26,7 @@
   >  
   >  import com.fasterxml.jackson.annotation.JsonIgnore;
   >  import com.google.common.collect.ImmutableSet;
   > +import com.google.protobuf.ByteString;
   >  import org.apache.hadoop.hdds.DatanodeVersion;
   >  import org.apache.hadoop.hdds.HddsUtils;
   >  import org.apache.hadoop.hdds.annotation.InterfaceAudience;
   > @@ -77,14 +78,33 @@ public static Codec<DatanodeDetails> getCodec() {
   >      return CODEC;
   >    }
   >  
   > +  static class StringWithByteString {
   > +    private final String string;
   > +    private final ByteString bytes;
   > +
   > +    StringWithByteString(String string) {
   > +      this.string = string;
   > +      this.bytes = ByteString.copyFromUtf8(string);
   > +    }
   > +
   > +    @Override
   > +    public String toString() {
   > +      return string;
   > +    }
   > +
   > +    ByteString getBytes() {
   > +      return bytes;
   > +    }
   > +  }
   > +
   >    /**
   >     * DataNode's unique identifier in the cluster.
   >     */
   >    private final UUID uuid;
   > -  private final String uuidString;
   > +  private final StringWithByteString uuidString;
   >    private final String threadNamePrefix;
   >  
   > -  private String ipAddress;
   > +  private StringWithByteString ipAddress;
   >    private String hostName;
   >    private final List<Port> ports;
   >    private String certSerialId;
   > @@ -100,9 +120,9 @@ public static Codec<DatanodeDetails> getCodec() {
   >    private DatanodeDetails(Builder b) {
   >      super(b.hostName, b.networkLocation, NetConstants.NODE_COST_DEFAULT);
   >      uuid = b.id;
   > -    uuidString = uuid.toString();
   > +    uuidString = new StringWithByteString(uuid.toString());
   >      threadNamePrefix = HddsUtils.threadNamePrefix(uuidString);
   > -    ipAddress = b.ipAddress;
   > +    ipAddress = new StringWithByteString(b.ipAddress);
   >      hostName = b.hostName;
   >      ports = b.ports;
   >      certSerialId = b.certSerialId;
   > @@ -127,7 +147,7 @@ public DatanodeDetails(DatanodeDetails datanodeDetails) {
   >          datanodeDetails.getParent(), datanodeDetails.getLevel(),
   >          datanodeDetails.getCost());
   >      this.uuid = datanodeDetails.uuid;
   > -    this.uuidString = uuid.toString();
   > +    this.uuidString = new StringWithByteString(uuid.toString());
   >      threadNamePrefix = HddsUtils.threadNamePrefix(uuidString);
   >      this.ipAddress = datanodeDetails.ipAddress;
   >      this.hostName = datanodeDetails.hostName;
   > @@ -161,7 +181,7 @@ public UUID getUuid() {
   >     * @return UUID of DataNode
   >     */
   >    public String getUuidString() {
   > -    return uuidString;
   > +    return uuidString.toString();
   >    }
   >  
   >    /**
   > @@ -170,7 +190,7 @@ public String getUuidString() {
   >     * @param ip IP Address
   >     */
   >    public void setIpAddress(String ip) {
   > -    this.ipAddress = ip;
   > +    this.ipAddress = new StringWithByteString(ip);
   >    }
   >  
   >    /**
   > @@ -179,7 +199,7 @@ public void setIpAddress(String ip) {
   >     * @return IP address
   >     */
   >    public String getIpAddress() {
   > -    return ipAddress;
   > +    return ipAddress.toString();
   >    }
   >  
   >    /**
   > @@ -429,7 +449,7 @@ public HddsProtos.DatanodeDetailsProto.Builder toProtoBuilder(
   >      builder.setUuid(getUuidString());
   >  
   >      if (ipAddress != null) {
   > -      builder.setIpAddress(ipAddress);
   > +      builder.setIpAddressBytes(ipAddress.getBytes());
   >      }
   >      if (hostName != null) {
   >        builder.setHostName(hostName);
   > ```
   
   Thanks for the suggestion, but I noticed that in OM, there are quite a few instances of `DatanodeDetail`, and this implementation adds more memory consumption to the member variable than before.
   
   The current my PR idea is not to add extra memory consumption, so it uses a global static Map to cache the `ByteString`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #6656:
URL: https://github.com/apache/ozone/pull/6656#issuecomment-2102808982

   @xichen01 , do you have comparison before and after this change?  What is the memory saving?  Is it in MBs or GBs?
   
   If there are many `DatanodeDetails` instances , we probably should cache `DatanodeDetails` instances  instead of the `ByteString` inside the `DatanodeDetails`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6656:
URL: https://github.com/apache/ozone/pull/6656#issuecomment-2102940045

   @szetszwo Thanks for your reply
   > If there are many DatanodeDetails instances , we probably should cache DatanodeDetails instances instead of the ByteString inside the DatanodeDetails.
   
   - But `DatanodeDetails` is not an immutable object, so it is not possible to simply cache `DatanodeDetails`, which would require considerable modification, and possibly have some compatibility issues.
   
   - Moreover, `DatanodeDetails` not only consumes memory space for its own objects, but also consumes a lot of resources during the encode/decode of `DatanodeDetails`, this PR is to reduce the consumption of `DatanodeDetails` encode.
   
   
   > @xichen01 , do you have comparison before and after this change? What is the memory saving? Is it in MBs or GBs?
   
   Here's an `alloc` flame graph that shows the effect of the optimization
   
   Before
   https://issues.apache.org/jira/secure/attachment/13068731/om-create-key-alloc-profiler-setxxxString.html
   After
   The `com/google/protobuf/ByteString.copyFromUtf8` in the `DatanodeDetail` disappear.
   https://issues.apache.org/jira/secure/attachment/13068732/om-create-key-alloc-profiler-setxxbytes.html
   
   
   Of course, this PR is only a small improvement, and it would be a big improvement to be able to cache the `DatanodeDetails`, which may require more work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6656:
URL: https://github.com/apache/ozone/pull/6656#issuecomment-2106237653

   @szetszwo Thanks for your new comments and patch, I have applied the patch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6656:
URL: https://github.com/apache/ozone/pull/6656#issuecomment-2103082479

   > This PR also adds Guava Cache with some hard coded size (50,000). ....
   
   In fact, the reason I use Guava Cache is to prevent accidents from occurring that would result in too many keys in the cache.  Guava Cache makes it easy to control how much to cache.
   
   If we can guarantee the number of keys to be cached ( hostname, ip, uuid, ... (should not be too much.)), I think we can use a simpler `Map` Cache structure such as `ConcurrentHashMap`
   
   > That's why I suggest to add StringWithByteString. Please consider it.
   
   `StringWithByteString` still needs to repeatedly calculate `ByteString.copyFromUtf8`, which moves the call of `ByteString.copyFromUtf8` from inside protobuf to `StringWithByteString` (Some `DatanodeDetails` have short lifecycles, may only be accessed once)
   But for `DatanodeDetail` in Pipeline Cache, it will be effective solution
   
   How do you think?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #6656:
URL: https://github.com/apache/ozone/pull/6656#discussion_r1597469945


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java:
##########
@@ -123,17 +125,17 @@ private DatanodeDetails(Builder b) {
   }
 
   public DatanodeDetails(DatanodeDetails datanodeDetails) {
-    super(datanodeDetails.getHostName(), datanodeDetails.getNetworkLocation(),
+    super(datanodeDetails.getHostNameAsByteString(), datanodeDetails.getNetworkLocationAsByteString(),
         datanodeDetails.getParent(), datanodeDetails.getLevel(),
         datanodeDetails.getCost());
     this.uuid = datanodeDetails.uuid;
-    this.uuidString = uuid.toString();
+    this.uuidString = new StringWithByteString(uuid.toString());

Review Comment:
   This should be:
   ```java
       this.uuidString = datanodeDetails.uuidString;
   ```



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/util/StringWithByteString.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util;
+
+import com.google.protobuf.ByteString;
+import net.jcip.annotations.Immutable;
+
+/**
+ * Class to encapsulate and cache the conversion of a Java String to a ByteString.
+ */
+@Immutable
+public final class StringWithByteString {
+  private final String string;
+  private final ByteString bytes;
+
+  public StringWithByteString(String string) {
+    this(string, ByteString.copyFromUtf8(string));
+  }
+
+  public StringWithByteString(String string, ByteString bytes) {
+    this.string = string;
+    this.bytes = bytes;
+  }
+
+  @Override
+  public String toString() {
+    return string;
+  }

Review Comment:
   My fault -- it is better to call it `getString` so that it is easier to trace.  Fortunately, IdealIJ can "rename only currently method" (not "rename base method").



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java:
##########
@@ -671,7 +727,7 @@ public Builder setNetworkName(String name) {
      * @return DatanodeDetails.Builder
      */
     public Builder setNetworkLocation(String loc) {
-      this.networkLocation = loc;
+      this.networkLocation = new StringWithByteString(NetUtils.normalize(loc));

Review Comment:
   Since the `loc` is likely to be already normalized, we may add another  `setNetworkLocation(String loc, ByteString locBytes)` method as below:
   ```java
       public Builder setNetworkLocation(String loc) {
         return setNetworkLocation(loc, null);
       }
   
       public Builder setNetworkLocation(String loc, ByteString locBytes) {
         final String normalized = NetUtils.normalize(loc);
         this.networkLocation = normalized.equals(loc) && locBytes != null
             ? new StringWithByteString(normalized, locBytes)
             : new StringWithByteString(normalized);
         return this;
       }
   ```



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/util/StringWithByteString.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util;
+
+import com.google.protobuf.ByteString;
+import net.jcip.annotations.Immutable;
+
+/**
+ * Class to encapsulate and cache the conversion of a Java String to a ByteString.
+ */
+@Immutable
+public final class StringWithByteString {
+  private final String string;
+  private final ByteString bytes;
+
+  public StringWithByteString(String string) {
+    this(string, ByteString.copyFromUtf8(string));
+  }
+
+  public StringWithByteString(String string, ByteString bytes) {
+    this.string = string;
+    this.bytes = bytes;

Review Comment:
   Check null:
   ```java
       this.string = Objects.requireNonNull(string, "string == null");
       this.bytes = Objects.requireNonNull(bytes, "bytes == null");
   ```



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeImpl.java:
##########
@@ -46,18 +47,22 @@ public class NodeImpl implements Node {
    * {@link NetConstants#PATH_SEPARATOR})
    * @param location this node's location
    */
-  public NodeImpl(String name, String location, int cost) {
-    if (name != null && name.contains(PATH_SEPARATOR_STR)) {
+  public NodeImpl(StringWithByteString name, StringWithByteString location, int cost) {
+    if (name != null && name.toString() != null && name.toString().contains(PATH_SEPARATOR_STR)) {

Review Comment:
   Let's enforce that `StringWithByteString.toString()` is never null?  We may add `Objects.requireNonNull(..)` in the `StringWithByteString` constructor.
   - If the original string is null, then `name == null`;
   - otherwise,  `name != null && name.toString() != null`.



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/protocol/DatanodeDetails.java:
##########
@@ -660,7 +704,19 @@ public Builder setHostName(String host) {
      * @return DatanodeDetails.Builder
      */
     public Builder setNetworkName(String name) {
-      this.networkName = name;
+      this.networkName = new StringWithByteString(name);
+      return this;
+    }

Review Comment:
   This is no longer used.  Let's remove it?



##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/util/StringWithByteString.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util;
+
+import com.google.protobuf.ByteString;
+import net.jcip.annotations.Immutable;
+
+/**
+ * Class to encapsulate and cache the conversion of a Java String to a ByteString.
+ */
+@Immutable
+public final class StringWithByteString {
+  private final String string;
+  private final ByteString bytes;
+
+  public StringWithByteString(String string) {
+    this(string, ByteString.copyFromUtf8(string));
+  }

Review Comment:
   For handling null, let's check it to
   ```java
     public static StringWithByteString valueOf(String string) {
       return string != null? new StringWithByteString(string, ByteString.copyFromUtf8(string)) : null;
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #6656:
URL: https://github.com/apache/ozone/pull/6656#issuecomment-2103109337

   > StringWithByteString still needs to repeatedly calculate ByteString.copyFromUtf8, which moves the call of ByteString.copyFromUtf8 from inside protobuf to StringWithByteString (Some DatanodeDetails have short lifecycles, may only be accessed once)
   
   Only the `DatanodeDetails` created from the `Builder` needs converting.  The `DatanodeDetails` created from another `DatanodeDetails` does not (updated the code in my previous comment).  The short lived `DatanodeDetails` probably are not created by the `Builder`.  No?
   
   BTW, we could also change `Builder` to pass `StringWithByteString` when converting proto-to-`DatanodeDetails`.  Then, it will already have the `ByteString`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-10811. Reduce UTF8 string encoding by caching encoding result [ozone]

Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #6656:
URL: https://github.com/apache/ozone/pull/6656#issuecomment-2103025390

   > ... `DatanodeDetails` is not an immutable object, ...
   
   SCM and other code probably don't needs different `DatanodeDetails` instances for the same datanode.   I agree that it is not easy to fix.  So, our change, although not fixing it, should not make it harder to fix it in the future.
   
   > ...  but also consumes a lot of resources during the encode/decode of DatanodeDetails, this PR is to reduce the consumption of DatanodeDetails encode.
   
   This PR also adds Guava `Cache` with some hard coded size (50,000).   Guava `Cache` is a complicated data structure.  It may run threads internally.  It needs cache lookup.  It will create more problems.  It does not worth to save the memory space for 50,000 short `ByteString`.
   
   That's why I suggest to add `StringWithByteString`.  Please consider it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org