[jira] [Created] (HDDS-2467) Allow running Freon validators with limited memory

["Attila Doroszlai (Jira)" <ji...@apache.org>]

Attila Doroszlai created HDDS-2467:
--------------------------------------

             Summary: Allow running Freon validators with limited memory
                 Key: HDDS-2467
                 URL: https://issues.apache.org/jira/browse/HDDS-2467
             Project: Hadoop Distributed Data Store
          Issue Type: Improvement
          Components: freon
            Reporter: Attila Doroszlai
            Assignee: Attila Doroszlai


Freon validators read each item to be validated completely into a {{byte[]}} buffer.  This allows timing only the read (and buffer allocation), but not the subsequent digest calculation.  However, it also means that memory required for running the validators is proportional to key size.

I propose to add a command-line flag to allow calculating the digest while reading the input stream.  This changes timing results a bit, since values will include the time required for digest calculation.  On the other hand, Freon will be able to validate huge keys with limited memory.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org