You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@iotdb.apache.org by "Song Ziyang (Jira)" <ji...@apache.org> on 2022/10/19 03:03:00 UTC

[jira] [Created] (IOTDB-4688) Streaming MD5 replace blocking MD5

Song Ziyang created IOTDB-4688:
----------------------------------

             Summary: Streaming MD5 replace blocking MD5
                 Key: IOTDB-4688
                 URL: https://issues.apache.org/jira/browse/IOTDB-4688
             Project: Apache IoTDB
          Issue Type: Sub-task
            Reporter: Song Ziyang


Currently in InstallSnapshot, statemachine has to pre-compute each file's md5 digest and provide it in {{{}FileInfo{}}}. When snapshot file is large, this computing is time-consuming and may cause leader timeout.
In this PR I propose to move the computing task along within the InstallSnapshot stream process. That is, for each SnapshotChunk, we can read the chunk & compute its moving MD5 digest in the same time. This can enable us to read file & compute file MD5 in one sequential scan and thus save IO costs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)