[jira] [Updated] (HDDS-13422) Ozone tool should preserve previous RocksDB options

["Ivan Andika (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ivan Andika updated HDDS-13422:
-------------------------------
    Summary: Ozone tool should preserve previous RocksDB options  (was: Offline RocksDB compact tool should preserve previous DBProfile configurations)

> Ozone tool should preserve previous RocksDB options
> ---------------------------------------------------
>
>                 Key: HDDS-13422
>                 URL: https://issues.apache.org/jira/browse/HDDS-13422
>             Project: Apache Ozone
>          Issue Type: Sub-task
>            Reporter: Ivan Andika
>            Assignee: Ivan Andika
>            Priority: Major
>              Labels: pull-request-available
>
> We tried to use the offline RocksDB compaction tool to delete all the tombstones from the keyTable in OM DB. We used RdbUtil#getLiveSSTFilesForCFs to check for the LiveFileMetadata for the SST files on keyTable and confirmed that the numDeletions are all zero, meaning that all tombstones have been compacted away. 
> However, we saw that the compacted OM DB result in larger number of SST files (14332 -> 16111) and larger size (800GB -> 1.1T). We found out that the OPTIONS files before compaction and after the compaction are different. Notably the block_size is reduced from 16KB to 4KB. This might explain why there are larger number of SST files (due to file splitting) and the size increase (due to lower compression ratio on the smaller blocks). 
> One suspect is that RocksDB#open will simply reinitialize the RocksDB options to the default options (block_size is 4KB by default). We should use RDBStore instead that will use the correct RocksDB configuration in DBProfile (DBProfile#getBlockBasedTableConfig). If this is correct, then this might affect other CLI as well (TransactionInfoRepair, SnapshotChainRepair).
> Update: From https://github.com/facebook/rocksdb/wiki/RocksDB-Options-File, we can use OptionUtil#loadLatestOptions and pass it during RocksDB open to use the latest options (i.e keep the options unchanged).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org