You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Sijie Guo (JIRA)" <ji...@apache.org> on 2012/11/06 15:38:12 UTC
[jira] [Created] (BOOKKEEPER-458) Annoy BKReadException error when
changing ledger.
Sijie Guo created BOOKKEEPER-458:
------------------------------------
Summary: Annoy BKReadException error when changing ledger.
Key: BOOKKEEPER-458
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
Project: Bookkeeper
Issue Type: Bug
Reporter: Sijie Guo
Priority: Minor
Fix For: 4.2.0
Some annoy BKReadException are found when changing ledger.
1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
2) T changed ledger to write entry to ledger L2.
3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
{code:title=BookKeeperPersistentManager.java}
// None of the old ledgers have this seq-id, we must use the
// current ledger
long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
+ topicInfo.lastEntryIdAckedInCurrentLedger;
if (endSeqId < startSeqId) {
request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
}
{code}
The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-458) Annoy BKReadException error when
changing ledger.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiannan Wang updated BOOKKEEPER-458:
------------------------------------
Component/s: hedwig-server
> Annoy BKReadException error when changing ledger.
> -------------------------------------------------
>
> Key: BOOKKEEPER-458
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Minor
> Fix For: 4.2.0
>
>
> Some annoy BKReadException are found when changing ledger.
> 1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
> 2) T changed ledger to write entry to ledger L2.
> 3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
> 4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
> {code:title=BookKeeperPersistentManager.java}
> // None of the old ledgers have this seq-id, we must use the
> // current ledger
> long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
> + topicInfo.lastEntryIdAckedInCurrentLedger;
> if (endSeqId < startSeqId) {
> request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
> }
> {code}
> The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
> The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-458) Annoy BKReadException error
when changing ledger.
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497322#comment-13497322 ]
Ivan Kelly commented on BOOKKEEPER-458:
---------------------------------------
This shouldn't be too hard to unit test. You need to construct a BookKeeperPersistenceManager (and unfortunately all the other things it needs), use #persistMessage a few times, and then call #changeLedger. A call to #scanMessages after that should trigger the bug.
> Annoy BKReadException error when changing ledger.
> -------------------------------------------------
>
> Key: BOOKKEEPER-458
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Minor
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-458.diff
>
>
> Some annoy BKReadException are found when changing ledger.
> 1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
> 2) T changed ledger to write entry to ledger L2.
> 3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
> 4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
> {code:title=BookKeeperPersistentManager.java}
> // None of the old ledgers have this seq-id, we must use the
> // current ledger
> long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
> + topicInfo.lastEntryIdAckedInCurrentLedger;
> if (endSeqId < startSeqId) {
> request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
> }
> {code}
> The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
> The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-458) Annoy BKReadException error
when changing ledger.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508266#comment-13508266 ]
Jiannan Wang commented on BOOKKEEPER-458:
-----------------------------------------
Yes, I plan to update the patch this week.
> Annoy BKReadException error when changing ledger.
> -------------------------------------------------
>
> Key: BOOKKEEPER-458
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Minor
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-458.diff
>
>
> Some annoy BKReadException are found when changing ledger.
> 1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
> 2) T changed ledger to write entry to ledger L2.
> 3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
> 4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
> {code:title=BookKeeperPersistentManager.java}
> // None of the old ledgers have this seq-id, we must use the
> // current ledger
> long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
> + topicInfo.lastEntryIdAckedInCurrentLedger;
> if (endSeqId < startSeqId) {
> request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
> }
> {code}
> The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
> The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-458) Annoy BKReadException error
when changing ledger.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504566#comment-13504566 ]
Hadoop QA commented on BOOKKEEPER-458:
--------------------------------------
Testing JIRA BOOKKEEPER-458
WARNING: Running test-patch on a dirty local svn workspace
> Annoy BKReadException error when changing ledger.
> -------------------------------------------------
>
> Key: BOOKKEEPER-458
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Minor
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-458.diff
>
>
> Some annoy BKReadException are found when changing ledger.
> 1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
> 2) T changed ledger to write entry to ledger L2.
> 3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
> 4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
> {code:title=BookKeeperPersistentManager.java}
> // None of the old ledgers have this seq-id, we must use the
> // current ledger
> long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
> + topicInfo.lastEntryIdAckedInCurrentLedger;
> if (endSeqId < startSeqId) {
> request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
> }
> {code}
> The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
> The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-458) Annoy BKReadException error
when changing ledger.
Posted by "Flavio Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508261#comment-13508261 ]
Flavio Junqueira commented on BOOKKEEPER-458:
---------------------------------------------
Hi Jiannan, I was wondering if you're planning on having a new patch uploaded soon.
> Annoy BKReadException error when changing ledger.
> -------------------------------------------------
>
> Key: BOOKKEEPER-458
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Minor
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-458.diff
>
>
> Some annoy BKReadException are found when changing ledger.
> 1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
> 2) T changed ledger to write entry to ledger L2.
> 3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
> 4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
> {code:title=BookKeeperPersistentManager.java}
> // None of the old ledgers have this seq-id, we must use the
> // current ledger
> long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
> + topicInfo.lastEntryIdAckedInCurrentLedger;
> if (endSeqId < startSeqId) {
> request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
> }
> {code}
> The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
> The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-458) Annoy BKReadException error when
changing ledger.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiannan Wang updated BOOKKEEPER-458:
------------------------------------
Attachment: BOOKKEEPER-458.diff
Attach a patch to fix the problem. Also post it to review board:
https://reviews.apache.org/r/8053/
Although the fix is very simple, it's difficult to design a test case. I think the reviewer can check it in mind easily :)
> Annoy BKReadException error when changing ledger.
> -------------------------------------------------
>
> Key: BOOKKEEPER-458
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Minor
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-458.diff
>
>
> Some annoy BKReadException are found when changing ledger.
> 1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
> 2) T changed ledger to write entry to ledger L2.
> 3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
> 4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
> {code:title=BookKeeperPersistentManager.java}
> // None of the old ledgers have this seq-id, we must use the
> // current ledger
> long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
> + topicInfo.lastEntryIdAckedInCurrentLedger;
> if (endSeqId < startSeqId) {
> request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
> }
> {code}
> The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
> The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-458) Annoy BKReadException error
when changing ledger.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497882#comment-13497882 ]
Jiannan Wang commented on BOOKKEEPER-458:
-----------------------------------------
I'll try to create a test case as your suggestion. Thanks Ivan.
> Annoy BKReadException error when changing ledger.
> -------------------------------------------------
>
> Key: BOOKKEEPER-458
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Minor
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-458.diff
>
>
> Some annoy BKReadException are found when changing ledger.
> 1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
> 2) T changed ledger to write entry to ledger L2.
> 3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
> 4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
> {code:title=BookKeeperPersistentManager.java}
> // None of the old ledgers have this seq-id, we must use the
> // current ledger
> long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
> + topicInfo.lastEntryIdAckedInCurrentLedger;
> if (endSeqId < startSeqId) {
> request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
> }
> {code}
> The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
> The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-458) Annoy BKReadException error when
changing ledger.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiannan Wang updated BOOKKEEPER-458:
------------------------------------
Assignee: Jiannan Wang
> Annoy BKReadException error when changing ledger.
> -------------------------------------------------
>
> Key: BOOKKEEPER-458
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-458
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Minor
> Fix For: 4.2.0
>
>
> Some annoy BKReadException are found when changing ledger.
> 1) suppose Topic T has ledger L1, storing messages starting from 1 - 100.
> 2) T changed ledger to write entry to ledger L2.
> 3) Before the entry is added successfully, Subscribe s subscribed topic T. ReadAhead cache tried to schedule a ReadAhead request to scan (103, 104).
> 4) RangeScanOp in BookKeeperPersistentManager executed to read entry 2 & 3 from L2. but actually there was no entries in L2.
> {code:title=BookKeeperPersistentManager.java}
> // None of the old ledgers have this seq-id, we must use the
> // current ledger
> long endSeqId = topicInfo.currentLedgerRange.getStartSeqIdIncluded()
> + topicInfo.lastEntryIdAckedInCurrentLedger;
> if (endSeqId < startSeqId) {
> request.getCallback().scanFinished(request.ctx, ReasonForFinish.NO_MORE_MESSAGES); return;
> }
> {code}
> The code in BookKeeperPersistentManager is supposed to not scan any messages whose seq id is larger than lastEntryIdAckedInCurrentLedger. But lastEntryIdAckedInCurrentLedger isn't reset when changing ledger. so when RangeScanOp is executed, last entry id acked in previous ledger was used which causing calculating an error seq id for the boundary checking in RangeScanOp.
> The fix would be quite easy to reset lastEntryIdAckedInCurrentLedger when changing ledger. But we need a test case to cover this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira