[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #152: HDDS-2467. Allow running Freon validators with limited memory

[GitBox <gi...@apache.org>]

adoroszlai opened a new pull request #152: HDDS-2467. Allow running Freon validators with limited memory
URL: https://github.com/apache/hadoop-ozone/pull/152
 
 
   ## What changes were proposed in this pull request?
   
   1. Freon validators read each item to be validated completely into a `byte[]` buffer.  This allows timing only the read (and buffer allocation), but not the subsequent digest calculation.  However, it also means that memory required for running the validators is proportional to key size.  I propose to add a command-line flag (`-s` / `--stream`) which, when specified, makes Freon calculate the digest while reading the input stream.  This changes timing results a bit, since values will include the time required for digest calculation.  On the other hand, Freon will be able to validate huge keys with limited memory.
   2. Reduce the memory requirement of the non-stream version by allocating a buffer exactly the size of the key.  This adds a bit of overhead in time, since key info needs to be fetched, too.  But it eliminates `ByteArrayOutputStream`, which allocates incrementally larger and larger buffers.  The latter can lead to memory requirement twice the actual key size in the worst case (since `2^n > 2^n-1 + 2^n-2 + ...`).
   3. Get rid of code duplication between `SameKeyReader` and `OzoneClientKeyValidator`.
   4. Allow `OzoneClientKeyGenerator` to create > 2GB keys.
   
   https://issues.apache.org/jira/browse/HDDS-2467
   
   ## How was this patch tested?
   
   Created and validated keys using Freon.  Verified that even 2.5GB key can be created and validated with `--stream`.  Verified that streaming is forced for such a large key, since it won't fit any array.  Verified that smaller keys can be validated both ways.
   
   ```
   export HADOOP_OPTS='-Xmx1024M -XX:+HeapDumpOnOutOfMemoryError'
   ozone freon ockg -t 1 -F ONE -n 1 -p 2_5GB -s 2684354560
   ozone freon ockg -t 1 -F ONE -n 1 -p 256MB -s  268435456
   ozone freon ockg -t 1 -F ONE -n 1 -p 128MB -s  134217728
   ozone freon ockg -t 1 -F ONE -n 1 -p  64MB -s   67108864
   ozone freon ockg -t 1 -F ONE -n 1 -p  10KB -s      10240
   
   export HADOOP_OPTS='-Xmx128M -XX:+HeapDumpOnOutOfMemoryError'
   ozone freon ockv -t 1 -n 1 -p  10KB
   ozone freon ockv -t 1 -n 1 -p  64MB
   
   export HADOOP_OPTS='-Xmx64M -XX:+HeapDumpOnOutOfMemoryError'
   ozone freon ockv -t 1 -n 1 -p  10KB -s
   ozone freon ockv -t 1 -n 1 -p  64MB -s
   ozone freon ockv -t 1 -n 1 -p 128MB -s
   ozone freon ockv -t 1 -n 1 -p 256MB -s
   ozone freon ockv -t 1 -n 1 -p 2_5GB -s
   ozone freon ockv -t 1 -n 1 -p 2_5GB
   
   ozone freon ockg -t 1 -F ONE -n 100 -p 1KB -s 1024
   ozone freon ockv -n 100 -p 1KB
   
   ozone freon ocokr -t 4 -k  '64MB/0' -n 32 -s
   ozone freon ocokr -t 8 -k '256MB/0' -n 16 -s
   
   export HADOOP_OPTS='-Xmx1024M -XX:+HeapDumpOnOutOfMemoryError'
   ozone freon ocokr -t 2 -k '256MB/0' -n 16
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org