[GitHub] rdhabalia opened a new pull request #1046: Introduce config to skip non-recoverable data-ledger

[GitBox <gi...@apache.org>]

rdhabalia opened a new pull request #1046: Introduce config to skip non-recoverable data-ledger
URL: https://github.com/apache/incubator-pulsar/pull/1046
 
 
   ### Motivation
   
   Recently, we have seen that due to unexpected ledger deletion at BK, Broker was seeing NoSuchEntryException/NoLedgerException while reading data-ledger.
   
   ```
   22:14:22.833 [BookKeeperClientWorker-20-1] WARN  o.a.b.mledger.impl.OpReadEntry       - [prop1/global/ns/persistent/topic1][sub1] read failed from ledger at position:5422597082:9745 : No such ledger exists
   22:14:33.806 [BookKeeperClientWorker-20-1] WARN  o.a.b.mledger.impl.OpReadEntry       - [prop1/global/ns/persistent/topic2][sub1] read failed from ledger at position:5422670450:5149 : No such entry
   :
   ```
   In this case, it requires manual cleanup to delete those ledgers from managed-ledger list and performing it for large number of topics across many brokers will be very complicated. In this case, those ledgers are not recoverable so, there should be a mechanism to skip non-recoverable data-ledgers from managed-ledger list.
   
   ### Modifications
   
   - Introduced a dynamic configuration to skip non-recoverable data-ledger
   - Skip reading non-recoverable data-ledger
   
   ### Result
   
   Broker will be more resilient when it is not able to read non-recoverable data-ledger and it can avoid manual cleanup in cluster.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services