[jira] [Assigned] (RATIS-1787) Don't generate md5 file for each file when a file is received during InstallSnapshot to reduce the pressure on the file system

["Xinyu Tan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/RATIS-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xinyu Tan reassigned RATIS-1787:
--------------------------------

    Assignee: Xinyu Tan

> Don't  generate md5 file for each file when a file is received during InstallSnapshot to reduce the pressure on the file system
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: RATIS-1787
>                 URL: https://issues.apache.org/jira/browse/RATIS-1787
>             Project: Ratis
>          Issue Type: Improvement
>            Reporter: Xinyu Tan
>            Assignee: Xinyu Tan
>            Priority: Major
>
> The IoTDB community has experienced a number of performance issues with MD5 calculations when using the Snapshot feature.
> Originally, MD5 calculation is to prevent errors in file transmission. However, in the previous implementation, when we generate a Snapshot, we need to load all files to calculate the snapshot, even though we do not need to send snapshot at this time, which causes a lot of resource preemption in the background. [~William Song] in RATIS-1597 has combined client-side MD5 calculations with streaming, thus avoiding a lot of background IO and computing tasks. However, at the Snapshot receiver, one MD5 file is still currently stored for each file. At present, the snapshot file level of IoTDB may be tens of thousands. These tens of thousands of small md5 files cause a lot of pressure on the file system. Is it possible that we do not store an md5 file for each file when receiving?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)