[GitHub] [bookkeeper] Vanlightly opened a new issue #2773: Rebuild db ledgers index command

[GitBox <gi...@apache.org>]

Vanlightly opened a new issue #2773:
URL: https://github.com/apache/bookkeeper/issues/2773


   File corruption of the ledgers index (DbLedgerStorage) has happened in production and we currently have no good way of resolving it. There exists a locations index rebuild command, this issue describes a new command targeted at the ledgers index.
   
   This command should only be run when either the bookie is offline, or when in readonly mode to avoid data loss. Container based environments may not be able to make the bookie go offline while also allowing the shell or bkctl to perform the operation.
   
   The db ledgers index rebuild op does the following:
   - moves the current index file to a backup location
   - scans all journal and entry log files to discover all ledger ids currently stored.
   - builds a new index where each ledger is fenced and has an empty master key set. 
   - The bookie should then be restarted again (in normal mode) to load the rebuilt index
   
   If a failure occurs during the operation, the original index file is restored to its original location. 
   
   Notable stuff:
   - The reason for setting fencing each ledger is that there is no safe way of setting the fenced status based on metadata while the bookie is running. If the command is run when the bookie is in readonly mode, it can still serve fence requests and any fencing that occurs while the rebuild is occuring will be lost, which breaks the data safety guarantees of the BookKeeper protocol. Given also that the bookie should at least be in readonly mode (else offline), it is likely a member of the current ensemble of few ledgers so the impact should be minimal.
   - The reason for setting an empty master key is that firstly, an empty master key simply gets overwritten, so cannot cause an IllegalOpException. Secondly, if we use the password stored in metadata, then we need to be sure to use exactly the same digest algorithm as the client when it creates the ledgerKey, else the bookie will start failing all writes. This could potentially cause a problem in the future if the way the ledgerKey is generated changes (old clients would be incompatible after a ledgers index rebuild).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bookkeeper] hsaputra closed issue #2773: Rebuild db ledgers index command

[GitBox <gi...@apache.org>]

hsaputra closed issue #2773:
URL: https://github.com/apache/bookkeeper/issues/2773


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org