You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Sammi Chen (Jira)" <ji...@apache.org> on 2022/10/13 07:44:00 UTC
[jira] [Created] (HDDS-7321) Support CLI command to trigger rocksDB range compaction
Sammi Chen created HDDS-7321:
--------------------------------
Summary: Support CLI command to trigger rocksDB range compaction
Key: HDDS-7321
URL: https://issues.apache.org/jira/browse/HDDS-7321
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Sammi Chen
Assignee: Sammi Chen
RocksDB has auto compaction itself, which is triggered by the total level file size.
Once the total level file size exceeds the threshold, RocksDB will schedule the compaction in backend.
When replicating containers between datanodes, current implementation leverages RocksDB SstFileWriter to export container meta data to individual sst files, and leverages RocksDB ingestExternalFile to import container meta data sst files directly into target datanode RocksDB. If the imported container meta data keys don't overlap with other sst files(Consider Merge RocksDB design, container ID is used as prefix of each meta data key, this is true for most of time), the imported sst file will be kept and remain the same without compacting with other existing sst files.
The worst case, if thousands or dozens of thousands of containers are imported on one datanode, there would be dozens of thousands of small sst files under one RocksDB, or across all RocksDB instances of one datanode. By default, RocksDB has no limit of open files. Dozens of thousands of small sst files would exhaust the process open file quota, and service stability will be impacted.
This task aims to provide an option for user to launch range compaction manually to enforce compact all these small sst files into merged big ones. Of course, the compaction will have impact on user data read/write performance on this datanode, so the suggestion is only triggering the operation when the datnode is not busy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org