LedgerRecovery seems doesn't work correctly when failure happened.

[Sijie Guo <gu...@gmail.com>]

A problem for ledger recovery is similar as
https://issues.apache.org/jira/browse/BOOKKEEPER-365 .

Before describing the issue, I need to clarify my understanding of ledger
recovery. After a ledger is closed, all the entries in the ledger should be
already ack'ed to a ledger writer. So we could provide a consistent view
for the readers and the writer. If there are entries existed in a closed
ledger don't ack to ledger writer, they should be invalid and break the
guarantee.

Currently recovery read logic is quite simple, if we read an available
entry from the quorum, we treated it as a valid entry. we don't check
enough bookies (ack_quorum_size) to ensure an entry is valid or not.

Still take an example,

A ledger with ensemble size 5, write quorum size 3, ack quorum size 2.
ensemble (a, b, c, d, e)

1. client issue 5 async add requests.

entry 0: a, b, c.
entry 1: b, c, d
entry 2: c, d, e
entry 3: d, e, a,
entry 4: e, a, b.

A, B, C succeed added all these 5 entries. But D, E failed.

2. so 0~1 are ack'ed to client, but 2~4 don't.

3. change ensemble kicked in to select new bookie to replace D and E.

4. but unfortunately client failed at this time.

5. a new client is elected to recover this ledger. base on current recover
read logic, all five entries would be treated as valid entries. it closed
the ledger with five entries 0~4. but from writer's perspective, it only
saw 0~1 are succeed. an inconsistent view introduced to clients.

Should we follow the ack_quorum_size semantic to check there are at least
{ack_quorum_size} replicas existed for an entry? similar as BOOKKEEPER-365.
The evidence of the existence of an entry is that we should receive a
return code which is neither NoSuchEntryException nor
BookieHandleNotAvailableException.

If it was a problem, it should be fixed in 4.2.0 as a blocker.

Re: LedgerRecovery seems doesn't work correctly when failure happened.

[Sijie Guo <gu...@gmail.com>]

The above example, client might not receive any response for 2~4.

replacing 4) as:

4' : there is not enough bookies available. so client tried to error out
2~4 and closed the ledger. application received the error callback for 2~4.
but it failed when closing the ledger.

after replacing 4 with 4', this example might be more convincible.


On Wed, Dec 12, 2012 at 11:34 PM, Sijie Guo <gu...@gmail.com> wrote:

> A problem for ledger recovery is similar as
> https://issues.apache.org/jira/browse/BOOKKEEPER-365 .
>
> Before describing the issue, I need to clarify my understanding of ledger
> recovery. After a ledger is closed, all the entries in the ledger should be
> already ack'ed to a ledger writer. So we could provide a consistent view
> for the readers and the writer. If there are entries existed in a closed
> ledger don't ack to ledger writer, they should be invalid and break the
> guarantee.
>
> Currently recovery read logic is quite simple, if we read an available
> entry from the quorum, we treated it as a valid entry. we don't check
> enough bookies (ack_quorum_size) to ensure an entry is valid or not.
>
> Still take an example,
>
> A ledger with ensemble size 5, write quorum size 3, ack quorum size 2.
> ensemble (a, b, c, d, e)
>
> 1. client issue 5 async add requests.
>
> entry 0: a, b, c.
> entry 1: b, c, d
> entry 2: c, d, e
> entry 3: d, e, a,
> entry 4: e, a, b.
>
> A, B, C succeed added all these 5 entries. But D, E failed.
>
> 2. so 0~1 are ack'ed to client, but 2~4 don't.
>
> 3. change ensemble kicked in to select new bookie to replace D and E.
>
> 4. but unfortunately client failed at this time.
>
> 5. a new client is elected to recover this ledger. base on current recover
> read logic, all five entries would be treated as valid entries. it closed
> the ledger with five entries 0~4. but from writer's perspective, it only
> saw 0~1 are succeed. an inconsistent view introduced to clients.
>
> Should we follow the ack_quorum_size semantic to check there are at least
> {ack_quorum_size} replicas existed for an entry? similar as BOOKKEEPER-365.
> The evidence of the existence of an entry is that we should receive a
> return code which is neither NoSuchEntryException nor
> BookieHandleNotAvailableException.
>
> If it was a problem, it should be fixed in 4.2.0 as a blocker.
>
>
>

Re: LedgerRecovery seems doesn't work correctly when failure happened.

[Sijie Guo <gu...@gmail.com>]

Hmm. If the contract is as what you described, we need to clarify it in
some documentation (I don't seem any documentation mention it. If I missed,
please point for me). otherwise, users might misuse it.

-Sijie


On Thu, Dec 13, 2012 at 12:33 AM, Flavio Junqueira <fp...@yahoo-inc.com>wrote:

> The contract of ledger recovery (and closing a ledger in general ) is that
> any entry that has been acknowledged to the application must be in the
> closed ledger. If entry x has been ack'ed and y is the last entry of the
> ledger according to the close procedure, then  x <= y.
>
> Ledger recovery, however, does not guarantee the converse. We might end up
> having entries in a closed ledger that haven't been ack'ed, and it is ok.
> Ledger recovery does guarantee that all entries have been correctly
> replicated, even the ones that might have been only partially replicated.
>
> -Flavio
>
> On Dec 13, 2012, at 8:34 AM, Sijie Guo wrote:
>
> A problem for ledger recovery is similar as
> https://issues.apache.org/jira/browse/BOOKKEEPER-365 .
>
> Before describing the issue, I need to clarify my understanding of ledger
> recovery. After a ledger is closed, all the entries in the ledger should be
> already ack'ed to a ledger writer. So we could provide a consistent view
> for the readers and the writer. If there are entries existed in a closed
> ledger don't ack to ledger writer, they should be invalid and break the
> guarantee.
>
> Currently recovery read logic is quite simple, if we read an available
> entry from the quorum, we treated it as a valid entry. we don't check
> enough bookies (ack_quorum_size) to ensure an entry is valid or not.
>
> Still take an example,
>
> A ledger with ensemble size 5, write quorum size 3, ack quorum size 2.
> ensemble (a, b, c, d, e)
>
> 1. client issue 5 async add requests.
>
> entry 0: a, b, c.
> entry 1: b, c, d
> entry 2: c, d, e
> entry 3: d, e, a,
> entry 4: e, a, b.
>
> A, B, C succeed added all these 5 entries. But D, E failed.
>
> 2. so 0~1 are ack'ed to client, but 2~4 don't.
>
> 3. change ensemble kicked in to select new bookie to replace D and E.
>
> 4. but unfortunately client failed at this time.
>
> 5. a new client is elected to recover this ledger. base on current recover
> read logic, all five entries would be treated as valid entries. it closed
> the ledger with five entries 0~4. but from writer's perspective, it only
> saw 0~1 are succeed. an inconsistent view introduced to clients.
>
> Should we follow the ack_quorum_size semantic to check there are at least
> {ack_quorum_size} replicas existed for an entry? similar as BOOKKEEPER-365.
> The evidence of the existence of an entry is that we should receive a
> return code which is neither NoSuchEntryException nor
> BookieHandleNotAvailableException.
>
> If it was a problem, it should be fixed in 4.2.0 as a blocker.
>
>
>
>

Re: LedgerRecovery seems doesn't work correctly when failure happened.

[Flavio Junqueira <fp...@yahoo-inc.com>]

The contract of ledger recovery (and closing a ledger in general ) is that any entry that has been acknowledged to the application must be in the closed ledger. If entry x has been ack'ed and y is the last entry of the ledger according to the close procedure, then  x <= y.

Ledger recovery, however, does not guarantee the converse. We might end up having entries in a closed ledger that haven't been ack'ed, and it is ok. Ledger recovery does guarantee that all entries have been correctly replicated, even the ones that might have been only partially replicated.

-Flavio

On Dec 13, 2012, at 8:34 AM, Sijie Guo wrote:

> A problem for ledger recovery is similar as https://issues.apache.org/jira/browse/BOOKKEEPER-365 .
> 
> Before describing the issue, I need to clarify my understanding of ledger recovery. After a ledger is closed, all the entries in the ledger should be already ack'ed to a ledger writer. So we could provide a consistent view for the readers and the writer. If there are entries existed in a closed ledger don't ack to ledger writer, they should be invalid and break the guarantee.
> 
> Currently recovery read logic is quite simple, if we read an available entry from the quorum, we treated it as a valid entry. we don't check enough bookies (ack_quorum_size) to ensure an entry is valid or not.
> 
> Still take an example,
> 
> A ledger with ensemble size 5, write quorum size 3, ack quorum size 2. ensemble (a, b, c, d, e)
> 
> 1. client issue 5 async add requests.
> 
> entry 0: a, b, c.
> entry 1: b, c, d
> entry 2: c, d, e
> entry 3: d, e, a,
> entry 4: e, a, b.
> 
> A, B, C succeed added all these 5 entries. But D, E failed.
> 
> 2. so 0~1 are ack'ed to client, but 2~4 don't.
> 
> 3. change ensemble kicked in to select new bookie to replace D and E.
> 
> 4. but unfortunately client failed at this time.
> 
> 5. a new client is elected to recover this ledger. base on current recover read logic, all five entries would be treated as valid entries. it closed the ledger with five entries 0~4. but from writer's perspective, it only saw 0~1 are succeed. an inconsistent view introduced to clients.
> 
> Should we follow the ack_quorum_size semantic to check there are at least {ack_quorum_size} replicas existed for an entry? similar as BOOKKEEPER-365. The evidence of the existence of an entry is that we should receive a return code which is neither NoSuchEntryException nor BookieHandleNotAvailableException. 
> 
> If it was a problem, it should be fixed in 4.2.0 as a blocker.
> 
>