[GitHub] ArvinDevel opened a new issue #942: BP-24: BookieScanner: Enhance Data Integrity

[GitBox <gi...@apache.org>]

ArvinDevel opened a new issue #942: BP-24: BookieScanner: Enhance Data Integrity
URL: https://github.com/apache/bookkeeper/issues/942
 
 
   BP
   
   This is the master ticket for tracking BP-24:
   
   Currently Bookie can't deal entry losing gracefully, the AutoRecovery is restricted to the bookie level, which means the AutoRecovery takes effect only after bookie is down. However when a disk fails, either or both the ledger index files and entry log files could potentially become corrupt. BookKeeper needs to provide mechanisms to identify and handle these problems.
   
   In this BP, we introduce Bookie Scanner, which is a background task, to scan index files and entry log files to detect possible corruptions. Since data corruption may happen at any time on any block on any Bookie, it is important to identify these errors in a timely manner. This way, the bookie can remove/compact corrupted entries and re-replicate entries from other replicas, to maintain data integrity and reduce client errors.
   
   Cutting this BP to two major task:
   
   - [ ] Scan Index File, add suspicious Ledgers and fix logic
   
   - [ ] Scan Entry Log, add suspicious EntryLogs and fix logic
   
   Proposal PR - #927 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services