You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Sammi Chen (Jira)" <ji...@apache.org> on 2022/05/23 08:59:00 UTC
[jira] [Updated] (HDDS-3630) Merge rocksdb in datanode
[ https://issues.apache.org/jira/browse/HDDS-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sammi Chen updated HDDS-3630:
-----------------------------
Description:
Currently, one rocksdb for one container. one container has 5GB capacity. 10TB data need more than 2000 rocksdb in one datanode. It's difficult to limit the memory of 2000 rocksdb. So maybe we should limited instance of rocksdb for each disk.
The design of improvement is in the follow link, but still is a draft.
TODO:
1. compatibility with current logic i.e. one rocksdb for each container
2. measure the memory usage before and after improvement
3. effect on efficiency of read and write.
[https://docs.google.com/document/d/18Ybg-NjyU602c-MYXaJHP6yrg-dVMZKGyoK5C_pp1mM/edit#]
Updated: 2021-10-29
We'll continue this topic with a new detailed design doc:
[https://docs.google.com/document/d/1hLtPJI8lSUhazMsoRKQuXuXs2gqixi6bQNTV9UnPsqs/edit?usp=sharing]
Updated: 2022-05-23
Refined the design doc:
[https://docs.google.com/document/d/1oPCRUVqTJDbYITbwvoVCOZ2-k108FwOXVhq9ukDKHbQ/edit?usp=sharing]
was:
Currently, one rocksdb for one container. one container has 5GB capacity. 10TB data need more than 2000 rocksdb in one datanode. It's difficult to limit the memory of 2000 rocksdb. So maybe we should limited instance of rocksdb for each disk.
The design of improvement is in the follow link, but still is a draft.
TODO:
1. compatibility with current logic i.e. one rocksdb for each container
2. measure the memory usage before and after improvement
3. effect on efficiency of read and write.
[https://docs.google.com/document/d/18Ybg-NjyU602c-MYXaJHP6yrg-dVMZKGyoK5C_pp1mM/edit#]
Updated: 2021-10-29
We'll continue this topic with a new detailed design doc:
[https://docs.google.com/document/d/1hLtPJI8lSUhazMsoRKQuXuXs2gqixi6bQNTV9UnPsqs/edit?usp=sharing]
> Merge rocksdb in datanode
> -------------------------
>
> Key: HDDS-3630
> URL: https://issues.apache.org/jira/browse/HDDS-3630
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: runzhiwang
> Assignee: Mark Gui
> Priority: Major
> Labels: pull-request-available
> Attachments: Merge RocksDB in Datanode-v1.pdf, Merge RocksDB in Datanode-v2.pdf, Merge rocksdb in datanode-v3.pdf, Merge rocksdb in datanode-v4.pdf
>
>
> Currently, one rocksdb for one container. one container has 5GB capacity. 10TB data need more than 2000 rocksdb in one datanode. It's difficult to limit the memory of 2000 rocksdb. So maybe we should limited instance of rocksdb for each disk.
> The design of improvement is in the follow link, but still is a draft.
> TODO:
> 1. compatibility with current logic i.e. one rocksdb for each container
> 2. measure the memory usage before and after improvement
> 3. effect on efficiency of read and write.
> [https://docs.google.com/document/d/18Ybg-NjyU602c-MYXaJHP6yrg-dVMZKGyoK5C_pp1mM/edit#]
>
> Updated: 2021-10-29
> We'll continue this topic with a new detailed design doc:
> [https://docs.google.com/document/d/1hLtPJI8lSUhazMsoRKQuXuXs2gqixi6bQNTV9UnPsqs/edit?usp=sharing]
>
> Updated: 2022-05-23
> Refined the design doc:
> [https://docs.google.com/document/d/1oPCRUVqTJDbYITbwvoVCOZ2-k108FwOXVhq9ukDKHbQ/edit?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org