You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org> on 2012/11/04 07:37:12 UTC
[jira] [Created] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Yixue (Andrew) Zhu created BOOKKEEPER-455:
---------------------------------------------
Summary: Bookie recovery made assumption that entries are persisted in entry id order
Key: BOOKKEEPER-455
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
Project: Bookkeeper
Issue Type: Bug
Components: bookkeeper-server
Affects Versions: 4.2.0
Reporter: Yixue (Andrew) Zhu
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yixue (Andrew) Zhu updated BOOKKEEPER-455:
------------------------------------------
Description:
The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494472#comment-13494472 ]
Yixue (Andrew) Zhu commented on BOOKKEEPER-455:
-----------------------------------------------
Thanks Sijie,
"I need to clarify that Hedwig actually uses async interface for add entry."
Yes, that is what I referred to as "(Hedwig uses single-queue per topic)".
After looking at code more, Flavio and your comments makes sense.
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490761#comment-13490761 ]
Yixue (Andrew) Zhu commented on BOOKKEEPER-455:
-----------------------------------------------
Sijie,
I understand what LAC is.
The description is that It is *NOT* used by bookie. The fencing description https://cwiki.apache.org/BOOKKEEPER/fencing.html does not match with actual implementation.
Instead, ledger recovery just look at the index file to get last-written-entry. It seems that an assumption is made that client will not send next entry until previous entry get ACKed.
The logic does not seem to be correct -
Last-written-entry-id is used as last-acked-entry-id.
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490407#comment-13490407 ]
Yixue (Andrew) Zhu commented on BOOKKEEPER-455:
-----------------------------------------------
BTW, the fencing description https://cwiki.apache.org/BOOKKEEPER/fencing.html does not match with actual implementation.
According to the description, it seems at one point, the last-confirmed-entry-id in the add-entry-package was used (returned during recovery).
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490272#comment-13490272 ]
Flavio Junqueira commented on BOOKKEEPER-455:
---------------------------------------------
Hi Andrew, There are a couple of things I'd like to clarify here. We use TCP, so I would expect requests to be delivered to each bookie in order, even if there are packet drops, retransmissions, and such. Now even if the entries are delivered out of order, using an older last-add-confirmed value (LAC) does not hurt correctness. The LAC value is just a hint. But, it is correct that getting a LAC value closer to the last confirmed will speed up recovery.
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490536#comment-13490536 ]
Sijie Guo commented on BOOKKEEPER-455:
--------------------------------------
{quote}
Suppose bookie persist entry 1, 3, 4, and entry 2 failed and being retried (TCP handles IP layer package lost), and now bookie crash.
If the ledger is recovered, last-confirmed-entry-id can be set to 4. When readers try to read 2, it will not get it.
{quote}
some explanations about LAC.
during initialization, -1 is LAC.
case1:
1) adding 0 : sending 0 with (LAC=-1)
2) acked 0 : LAC changed from -1 to 0
3) adding 1 : sending 1 with (LAC=0)
4) acked 1 : LAC changed from 0 to 1
case2:
1) adding 0, 1, 2, 3, 4 : sending 0, 1, 2, 3, 4 with (LAC=-1)
2) acked 0: LAC changed from -1 to 0
3) acked 1: LAC changed from 0 to 1
4) failed to add 2.
in this case, LAC is -1 since the hint stored with 0~4 are -1. and LAC would not be larger than the entry that isn't ACKed.
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yixue (Andrew) Zhu resolved BOOKKEEPER-455.
-------------------------------------------
Resolution: Not A Problem
The issue impacts potential performance, but no functional problem.
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490290#comment-13490290 ]
Yixue (Andrew) Zhu commented on BOOKKEEPER-455:
-----------------------------------------------
Flavio,
First, I do not see the LAC field of add-entry-package is used by bookie. So, I am not concerned about that.
The issue is that Bookie client API asyncAddEntry send entries asynchronously. The bookie just persist what is requested by the client.
Suppose bookie persist entry 1, 3, 4, and entry 2 failed and being retried (TCP handles IP layer package lost), and now bookie crash.
If the ledger is recovered, last-confirmed-entry-id can be set to 4. When readers try to read 2, it will not get it.
The API is not used by production code (only by test).
The wrapper of the API, such as addEntry, wait until previous entry get acknowledged by Bookie.
The bookie server could enforce this behavior by checking the last-confirmed-entry-id in the add-entry-package, rejecting package where the last-confirmed-entry-id != entry_id - 1. Thought?
I will change the issue to be minor/cleanup, since no production problem is caused by it.
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491230#comment-13491230 ]
Sijie Guo commented on BOOKKEEPER-455:
--------------------------------------
{quote}
According to the description, it seems at one point, the last-confirmed-entry-id in the add-entry-package was used
{quote}
{quote}
Instead, ledger recovery just look at the index file to get last-written-entry.
{quote}
As the code below, Bookie actually responds last written entry. But in BookKeeper client, ReadLastConfirmedOp would try to read LAC carried in the last written entry. LAC is retrieved here when fencing read -1. The client doesn't use last written entry id as last acked entry id. so the implementation is same as the documentation.
{code:title=InterleavedLedgerStorage.java}
/*
* If entryId is BookieProtocol.LAST_ADD_CONFIRMED, then return the last written.
*/
if (entryId == BookieProtocol.LAST_ADD_CONFIRMED) {
entryId = ledgerCache.getLastEntry(ledgerId);
}
{code}
{code:title=ReadLastConfirmedOp.java}
RecoveryData recoveryData = lh.macManager.verifyDigestAndReturnLastConfirmed(buffer); if (recoveryData.lastAddConfirmed > maxRecoveredData.lastAddConfirmed) {
maxRecoveredData = recoveryData;
}
{code}
{code:title=DigestManager.java}
RecoveryData verifyDigestAndReturnLastConfirmed(ChannelBuffer dataReceived) throws BKDigestMatchException {
verifyDigest(dataReceived);
dataReceived.readerIndex(8);
dataReceived.readLong(); // skip unused entryId
long lastAddConfirmed = dataReceived.readLong();
long length = dataReceived.readLong();
return new RecoveryData(lastAddConfirmed, length);
}
{code}
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yixue (Andrew) Zhu updated BOOKKEEPER-455:
------------------------------------------
Priority: Minor (was: Major)
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Yixue (Andrew) Zhu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491674#comment-13491674 ]
Yixue (Andrew) Zhu commented on BOOKKEEPER-455:
-----------------------------------------------
Sijie,
You are right that bookie client read LAC from last-written entry.
However, in the extreme case,
if the last-written entry carries LAC 7, while some previous written entry carries LAC 8. would that cause issue?
Again, I do not see Async-add-entry directly used by production code, so it is not an *real* issue per se. (Hedwig uses single-queue per topic)
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-455) Bookie recovery made assumption
that entries are persisted in entry id order
Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491963#comment-13491963 ]
Sijie Guo commented on BOOKKEEPER-455:
--------------------------------------
{quote}
if the last-written entry carries LAC 7, while some previous written entry carries LAC 8. would that cause issue?
{quote}
since LAC only changed only an entry is Acked, and BookKeeper preserves the Acked order submitted back to client. so if writing K entry with LAC L (L <= K-1), writing K+1 entry whose LAC would not be larger than K. so the case you described might not happen.
{quote}
I do not see Async-add-entry directly used by production code, so it is not an real issue per se. (Hedwig uses single-queue per topic)
{quote}
I need to clarify that Hedwig actually uses async interface for add entry. You could refer PersistOp in BookKeeperPersistenceManager, which runs asyncAddEntry in a synchronous op. The benefit of asyncAddEntry is performance. The downside of asyncAddEntry is we might get a LAC far away from the last confirmed when the client issues asyncAddEntry quickly, which would make the ledger recovery longer when failure, as Flavio indicated in his first comment.
> Bookie recovery made assumption that entries are persisted in entry id order
> ----------------------------------------------------------------------------
>
> Key: BOOKKEEPER-455
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-455
> Project: Bookkeeper
> Issue Type: Bug
> Components: bookkeeper-server
> Affects Versions: 4.2.0
> Reporter: Yixue (Andrew) Zhu
> Priority: Minor
>
> The entries sent by bookkeeper client can be out of order (due to network). The client uses queue to track last-confirmed-entry-id correctly.
> The bookie server will happily persist the entries out of order, i.e. gaps are possible. If bookie crash, the recovery can cause last-confirmed-entry-id to be set to the last-entry persisted by bookie. The gap is not tracked nor detected at server side.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira