You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Tsz-wo Sze (Jira)" <ji...@apache.org> on 2023/03/22 08:50:00 UTC

[jira] [Updated] (HDDS-8128) Deduplicate the ops in RDBBatchOperation

     [ https://issues.apache.org/jira/browse/HDDS-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz-wo Sze updated HDDS-8128:
-----------------------------
    Component/s: db
                     (was: OM)
    Description: 
In a multipart upload test, the key "testKey" had 1000-parts with 8KB each.  The same key was uploaded 10 times sequentially (i.e. it overwrote the previous upload) in a newly formatted cluster.  The replication was 3, so the total raw size of the key is ~ 24 MB.  After the test has completed, OM rocks db uses ~ 7.5 GB.

In this JIRA, we add a cache to RDBBatchOperation for deduplication.  Within a batch, the put-ops and delete-ops of the same key can be safely deduplicated.  Only the last op has to be applied to the db.  All the previous ops can be discarded.

  was:In a multipart upload test, the key "testKey" had 1000-parts with 8KB each.  The same key was uploaded 10 times sequentially (i.e. it overwrote the previous upload) in a newly formatted cluster.  The replication was 3, so the total raw size of the key is ~ 24 MB.  After the test has completed, OM rocks db uses ~ 7.5 GB.

        Summary: Deduplicate the ops in RDBBatchOperation  (was: OM rocksdb uses a lot of space)

In this JIRA, we will focus on RDBBatchOperation deduplication, where RDBBatchOperation is a utility class used everywhere including OM, SCM, DN, etc.

Filed HDDS-8238 for some further works specific OM.

> Deduplicate the ops in RDBBatchOperation
> ----------------------------------------
>
>                 Key: HDDS-8128
>                 URL: https://issues.apache.org/jira/browse/HDDS-8128
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: db
>            Reporter: Tsz-wo Sze
>            Assignee: Tsz-wo Sze
>            Priority: Blocker
>              Labels: pull-request-available
>
> In a multipart upload test, the key "testKey" had 1000-parts with 8KB each.  The same key was uploaded 10 times sequentially (i.e. it overwrote the previous upload) in a newly formatted cluster.  The replication was 3, so the total raw size of the key is ~ 24 MB.  After the test has completed, OM rocks db uses ~ 7.5 GB.
> In this JIRA, we add a cache to RDBBatchOperation for deduplication.  Within a batch, the put-ops and delete-ops of the same key can be safely deduplicated.  Only the last op has to be applied to the db.  All the previous ops can be discarded.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org