[GitHub] [hadoop-ozone] anuengineer commented on a change in pull request #152: HDDS-2467. Allow running Freon validators with limited memory

[GitBox <gi...@apache.org>]

anuengineer commented on a change in pull request #152: HDDS-2467. Allow running Freon validators with limited memory
URL: https://github.com/apache/hadoop-ozone/pull/152#discussion_r346449625
 
 

 ##########
 File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/OzoneClientKeyValidator.java
 ##########
 @@ -80,16 +91,74 @@ public Void call() throws Exception {
     return null;
   }
 
+  private void readReference() throws IOException {
+    String name = generateObjectName(0);
+
+    if (!stream) {
+      // first obtain key size to be able to allocate exact buffer for keys
+      referenceKeySize = getKeySize(name);
+
+      // force stream if key is too large for byte[]
+      // (limit taken from ByteArrayOutputStream)
+      if (referenceKeySize > Integer.MAX_VALUE - 8) {
 
 Review comment:
   You can force this much sooner, say 1 GB ? or even 100MB. Otherwise, the virtual memory reservation would be huge.
   And practically, does it matter?  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org