[GitHub] [ozone] smengcl commented on pull request #3824: HDDS-7281. [Snapshot] Handle RocksDB compaction DAG persistence and reconstruction

[GitBox <gi...@apache.org>]

smengcl commented on PR #3824:
URL: https://github.com/apache/ozone/pull/3824#issuecomment-1276821184

   > This is in response to the discussion here: [#3786 (comment)](https://github.com/apache/ozone/pull/3786#discussion_r992935645)
   > 
   > To me there is value in the conceptual simplicity of having all the state in rocksdb.
   > 
   > And I don't think it would have to be a new RDB instance. It could just be a separate CF in the existing one. I know that that CF wouldn't be shared through ratis transactions, but it would be shared with followers when they get bootstrapped (which happens to be what we want.)
   
   Doesn't bootstrapping pack the entire `ozone-metadata` directory? If this is the case then the new `compaction-log` directory would have been included automatically. (I need to double check.)
   
   I understand it would be easier (and cleaner) if we just use a new CF in the **existing** OM RDB to store compaction history. The implementation would be trivial. But there is some aspects I dislike about it:
   
   1. The info we are storing to the DB (input / output SST file pairs) tightly relates to the file structure of the database itself, which isn't really related to main functions of OM, and can differ from OM to OM. Plus the fact that the compaction log isn't crucial for the RocksDiff feature to work (as it is meant for acceleration by returning only the relevant set of SST files). If we do want to persist the log in RDB I would prefer starting a new RDB instance just for this (a metadata DB of the existing DB).
   2. We are putting extra pressure on RDB compaction by writing those extra K-V pairs to the DB **itself** after each compaction. Though the pressure should be small provided that compaction doesn't happen too frequently (depends on DB write).
   
   > Finally, with respect to the ordering problem mentioned above, couldn't the db lastSeqNum be used as the rocksdb key, (as it is used in the filename currently)?
   
   Good point.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org