[jira] [Comment Edited] (BOOKKEEPER-447) Bookie can fail to recover if index pages flushed before ledger flush acknowledged

["Sijie Guo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BOOKKEEPER-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13530721#comment-13530721 ] 

Sijie Guo edited comment on BOOKKEEPER-447 at 12/13/12 6:21 AM:
----------------------------------------------------------------

Although this fix is quite simple and straightforward, I still could not be convinced using this solution. Because all writing requests are blocked until a flush is scheduled by SyncThread, so the latency heavily depends on the *flushInterval*. Also, it just resolve the problem force flushing ledger when grabbing a clean page. but if sync thread happens to flush the ledger before journal flushes, same problem occurred. 

the right direction resolving this issue is that we should guarantee journal flushing before ledger index flushed. A possible way is to add a listener on Journal, when it flushed entires, it notified LedgerStorage which ledger is flushed. so LedgerStorage could know which ledger page it should flush, which page it should not flush.

so the waiting time to grab a clean page is different for different solution, either depends on journal flushing latency or sync thread flush interval. journal flushing latency supposed to be very very low since it was used as a separated disk. but sync thread flush interval depends on user's setting and also it was a shared disk with read requests.


                
      was (Author: hustlmsp):
    Although this fix is quite simple and straightforward, I still could not be convinced using this solution. Because all writing requests are blocked until a flush is scheduled by SyncThread, so the latency heavily depends on the *flushInterval*. Also, it just resolve the problem force flushing ledger when grabbing a clean page. but if sync thread happens to flush the ledger before journal flushes, same problem occurred. 

the right direction resolving this issue is that we should guarantee journal flushing before ledger index flushed. A possible way is to add a listener on Journal, when it flushed entires, it notified LedgerStorage which ledger is flushed. so LedgerStorage could know which ledger page it should flush, which page it should not flush.

so the waiting time to grab a clean page is journal flushing latency or sync thread flush interval. journal flushing latency supposed to be very very low since it was used as a separated disk. but sync thread flush interval depends on user's setting and also it was a shared disk with read requests.


                  
> Bookie can fail to recover if index pages flushed before ledger flush acknowledged
> ----------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-447
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-447
>             Project: Bookkeeper
>          Issue Type: Bug
>          Components: bookkeeper-server
>    Affects Versions: 4.2.0
>            Reporter: Yixue (Andrew) Zhu
>            Assignee: Ivan Kelly
>             Fix For: 4.2.0, 4.1.1
>
>         Attachments: 0001-BOOKKEEPER-447-LedgerCacheImpl-waits-on-lock-object-.patch, 0001-BOOKKEEPER-447-LedgerCacheImpl-waits-on-semaphore-no.patch, BOOKKEEPER-447.diff, perf.png
>
>
> Bookie index page steal (LedgerCacheImpl::grabCleanPage) can cause index file to reflect unacknowledged entries (due to flushLedger). Suppose ledger and entry fail to flush due to Bookkeeper server crash, it will cause ledger recovery not able to use the bookie afterward, due to InterleavedStorageLedger::getEntry throws IOException.
> If the ackSet bookies all experience this problem (DC environment), the ledger will not be able to recover.
> The problem here essentially a violation of WAL. One reasonable fix is to track ledger flush progress (either per-ledger entry, or per-topic message). Do not flush index pages which tracks entries whose ledger (log) has not been flushed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira