[jira] [Commented] (BOOKKEEPER-365) Ledger will never recover if one of the quorum bookie is down forever and others dont have entry

["Flavio Junqueira (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BOOKKEEPER-365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13529078#comment-13529078 ] 

Flavio Junqueira commented on BOOKKEEPER-365:
---------------------------------------------

Let me add a few comments to this discussion.

In Sijie's example, has the entry is being read ever written successfully to all of A, B, and C? From the example, it is not clear, but I'm assuming that the write quorum is 3. If so, then it violates an assumption we make for the system. We can detect corrupt entries with the digest, but we can't really always guarantee recovery in the case that bookies drop entries altogether. 

To simplify perhaps the discussion and illustrate what we guarantee, say that we write entries to two bookies. When it is time to read a given entry (successfully written) let's say that one bookie has corrupted it and the other is partitioned away. The client executing the recovery procedure see the corrupt copy of the entry and it doesn't know if the other bookie has a good copy or not. If it does, then the entry has been written successfully. If it doesn't, then the entry hasn't been written successfully. To decide, we need the second bookie, so we can't close the ledger until the second bookie comes back. We should perhaps review what exception we would get in this case. 

In Andrew's observation, it is correct that if the client hasn't written enough copies (ack quorum), then we don't guarantee that the entry will be there when the ledger is closed. Keep in mind that we don't use all possible read/write quorums, since we pick quorums in a round-robin fashion. When recovering a ledger, we need to touch every write quorum we can possibly use to fence off the writer. Read and write quorums overlap fully by design. 

One point I don't understand is why we should expect to get the same LAC (Last Add Confirmed) from two different bookies. Each bookie could have a different one, and it would be ok. Also, keep in mind that LAC is only a hint. After we pick a LAC, we still try to read after that value to see how far we can go.

Finally, it would be good to decide if there is anything here to fix for 4.2.0. I'm smelling that there isn't, but I'd like to hear opinions.
                
> Ledger will never recover if one of the quorum bookie is down forever and others dont have entry
> ------------------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-365
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-365
>             Project: Bookkeeper
>          Issue Type: Bug
>    Affects Versions: 4.0.0, 4.1.0
>            Reporter: Sijie Guo
>            Assignee: Yixue (Andrew) Zhu
>             Fix For: 4.2.0
>
>
> As discussed in BOOKKEEPER-355, current fix to handle the below issue is not correct. Need to find out new solution
> If some bookies of a quorum gone forever, some bookies of this quorum are still alive but doesn't have that entry (NoSuchEntry or NoSuchLedger), then the ledger doesn't have any evidence to recovery/close it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira