You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Abhishek Pal (Jira)" <ji...@apache.org> on 2026/02/18 20:44:00 UTC

[jira] [Assigned] (HDDS-8238) Multipart upload generates a huge OM rocksdb log

     [ https://issues.apache.org/jira/browse/HDDS-8238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abhishek Pal reassigned HDDS-8238:
----------------------------------

    Assignee: Abhishek Pal

> Multipart upload generates a huge OM rocksdb log
> ------------------------------------------------
>
>                 Key: HDDS-8238
>                 URL: https://issues.apache.org/jira/browse/HDDS-8238
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: OM
>            Reporter: Tsz-wo Sze
>            Assignee: Abhishek Pal
>            Priority: Major
>              Labels: ozone-performance
>
> When using multipart upload to upload a 1000-part key. OM RocksDb log needs to use 733.15 MB disk space. For a 10,000-part key, it can use 71.5 GB .  The disk space usage for the log is O(n^2) for a n-part key.
> MultipartKeyInfo has a list of PartKeyInfo (partKeyInfoList) while each PartKeyInfo has a KeyInfo and the size of each KeyInfo is ~1.5 KB.
> {code:java}
> //OmClientProtocol.proto
> message MultipartKeyInfo {
>     required string uploadID = 1;
>     ...
>     repeated PartKeyInfo partKeyInfoList = 5;
>     ...
> }
> message PartKeyInfo {
>     required string partName = 1;
>     required uint32 partNumber = 2;
>     required KeyInfo partKeyInfo = 3; // the size of a KeyInfo is ~1.5 KB
> }
> {code}
> Multipart upload repeatedly puts the same key with a MultipartKeyInfo object for each part.
>  * When uploading the first part, partKeyInfoList has 1 element.
>  * When uploading the second part, partKeyInfoList has 2 elements.
>  * When uploading the third part, partKeyInfoList has 3 elements.
> ...
>  * When uploading the 1000th part, partKeyInfoList has 1000 elements.
> The partKeyInfoList is written to RocksDb log for each part. The number of PartKeyInfo objects written to RocksDb log is
>  - (1 + 1000)*1000/2 = 500,500 objects
> and the size is 1.5 KB * 500,500 = 733.15 MB.
> Note that only the RocksDb log size is large.  The size of the RocksDb tables are small since the partKeyInfoList will be overwritten again and again for each part.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org