[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #152: HDDS-2467. Allow running Freon validators with limited memory

[GitBox <gi...@apache.org>]

adoroszlai commented on a change in pull request #152: HDDS-2467. Allow running Freon validators with limited memory
URL: https://github.com/apache/hadoop-ozone/pull/152#discussion_r346463602
 
 

 ##########
 File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyValidator.java
 ##########
 @@ -80,16 +91,74 @@ public Void call() throws Exception {
     return null;
   }
 
+  private void readReference() throws IOException {
+    String name = generateObjectName(0);
+
+    if (!stream) {
+      // first obtain key size to be able to allocate exact buffer for keys
+      referenceKeySize = getKeySize(name);
+
+      // force stream if key is too large for byte[]
+      // (limit taken from ByteArrayOutputStream)
+      if (referenceKeySize > Integer.MAX_VALUE - 8) {
 
 Review comment:
   Any lower limit would be arbitrary.  Actual memory requirement depends on the number of threads.  One can always add the flag explicitly.  I just wanted to preserve existing behavior as a courtesy.
   
   Hope that makes sense :)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org