You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Sijie Guo (JIRA)" <ji...@apache.org> on 2012/10/25 07:12:12 UTC
[jira] [Created] (BOOKKEEPER-442) Failed to deliver messages due to
inconsistency between SubscriptionState and LedgerRanges.
Sijie Guo created BOOKKEEPER-442:
------------------------------------
Summary: Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
Key: BOOKKEEPER-442
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
Project: Bookkeeper
Issue Type: Bug
Components: hedwig-server
Affects Versions: 4.1.0, 4.2.0
Reporter: Sijie Guo
Assignee: Jiannan Wang
Priority: Critical
Fix For: 4.2.0
The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
The issue is described as below:
1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
{code}
AbstractSubscriptionManager:
if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
}
{code}
2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
{code}
public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
if (interval <= 0) {
return false;
}
// set consume seq id when it is larger
this.lastConsumeSeqId = lastConsumeSeqId;
if (interval < consumeInterval) {
return false;
}
// subscription state will be updated, marked it as clean
subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
return true;
}
{code}
3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
{code}
for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
}
{code}
The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-442) Failed to deliver messages due to
inconsistency between SubscriptionState and LedgerRanges.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiannan Wang updated BOOKKEEPER-442:
------------------------------------
Attachment: BOOKKEEPER-442.diff
Change follows Ivan's comment: logging the error instead of throw a runtime exception
> Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
> -------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-442
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-442.diff, BOOKKEEPER-442.diff
>
>
> The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
> The issue is described as below:
> 1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
> {code}
> AbstractSubscriptionManager:
>
> if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
> }
> {code}
> 2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
> {code}
> public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
> long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
> if (interval <= 0) {
> return false;
> }
> // set consume seq id when it is larger
> this.lastConsumeSeqId = lastConsumeSeqId;
> if (interval < consumeInterval) {
> return false;
> }
> // subscription state will be updated, marked it as clean
> subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
> return true;
> }
> {code}
> 3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
> {code}
> for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
> if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
> minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
> hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
> }
> {code}
> The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-442) Failed to deliver messages due
to inconsistency between SubscriptionState and LedgerRanges.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504605#comment-13504605 ]
Hadoop QA commented on BOOKKEEPER-442:
--------------------------------------
Testing JIRA BOOKKEEPER-442
WARNING: Running test-patch on a dirty local svn workspace
Patch <a href="/jira/secure/attachment/12552085/BOOKKEEPER-442.diff">/jira/secure/attachment/12552085/BOOKKEEPER-442.diff</a> downloaded at Tue Nov 27 13:17:20 UTC 2012
----------------------------
{color:red}-1{color} Patch failed to apply to head of branch
----------------------------
> Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
> -------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-442
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-442.diff, BOOKKEEPER-442.diff
>
>
> The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
> The issue is described as below:
> 1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
> {code}
> AbstractSubscriptionManager:
>
> if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
> }
> {code}
> 2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
> {code}
> public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
> long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
> if (interval <= 0) {
> return false;
> }
> // set consume seq id when it is larger
> this.lastConsumeSeqId = lastConsumeSeqId;
> if (interval < consumeInterval) {
> return false;
> }
> // subscription state will be updated, marked it as clean
> subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
> return true;
> }
> {code}
> 3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
> {code}
> for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
> if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
> minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
> hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
> }
> {code}
> The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-442) Failed to deliver messages due to
inconsistency between SubscriptionState and LedgerRanges.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiannan Wang updated BOOKKEEPER-442:
------------------------------------
Attachment: BOOKKEEPER-442.diff
https://reviews.apache.org/r/7775
Diff revision 1
> Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
> -------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-442
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-442.diff
>
>
> The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
> The issue is described as below:
> 1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
> {code}
> AbstractSubscriptionManager:
>
> if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
> }
> {code}
> 2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
> {code}
> public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
> long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
> if (interval <= 0) {
> return false;
> }
> // set consume seq id when it is larger
> this.lastConsumeSeqId = lastConsumeSeqId;
> if (interval < consumeInterval) {
> return false;
> }
> // subscription state will be updated, marked it as clean
> subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
> return true;
> }
> {code}
> 3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
> {code}
> for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
> if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
> minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
> hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
> }
> {code}
> The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-442) Failed to deliver messages due
to inconsistency between SubscriptionState and LedgerRanges.
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489371#comment-13489371 ]
Ivan Kelly commented on BOOKKEEPER-442:
---------------------------------------
Patch looks good. I have one question about the point where you throw a runtime exception, but otherwise this is ready to go.
> Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
> -------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-442
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-442.diff
>
>
> The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
> The issue is described as below:
> 1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
> {code}
> AbstractSubscriptionManager:
>
> if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
> }
> {code}
> 2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
> {code}
> public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
> long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
> if (interval <= 0) {
> return false;
> }
> // set consume seq id when it is larger
> this.lastConsumeSeqId = lastConsumeSeqId;
> if (interval < consumeInterval) {
> return false;
> }
> // subscription state will be updated, marked it as clean
> subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
> return true;
> }
> {code}
> 3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
> {code}
> for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
> if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
> minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
> hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
> }
> {code}
> The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-442) Failed to deliver messages due
to inconsistency between SubscriptionState and LedgerRanges.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509604#comment-13509604 ]
Hudson commented on BOOKKEEPER-442:
-----------------------------------
Integrated in bookkeeper-trunk #837 (See [https://builds.apache.org/job/bookkeeper-trunk/837/])
BOOKKEEPER-442: Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges. (jiannan via ivank) (Revision 1416560)
Result = UNSTABLE
ivank :
Files :
* /zookeeper/bookkeeper/trunk/CHANGES.txt
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/persistence/BookkeeperPersistenceManager.java
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/subscriptions/AbstractSubscriptionManager.java
* /zookeeper/bookkeeper/trunk/hedwig-server/src/main/java/org/apache/hedwig/server/subscriptions/InMemorySubscriptionState.java
* /zookeeper/bookkeeper/trunk/hedwig-server/src/test/java/org/apache/hedwig/server/persistence/TestBookKeeperPersistenceManager.java
> Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
> -------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-442
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-442.diff, BOOKKEEPER-442.diff, BOOKKEEPER-442.diff
>
>
> The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
> The issue is described as below:
> 1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
> {code}
> AbstractSubscriptionManager:
>
> if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
> }
> {code}
> 2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
> {code}
> public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
> long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
> if (interval <= 0) {
> return false;
> }
> // set consume seq id when it is larger
> this.lastConsumeSeqId = lastConsumeSeqId;
> if (interval < consumeInterval) {
> return false;
> }
> // subscription state will be updated, marked it as clean
> subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
> return true;
> }
> {code}
> 3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
> {code}
> for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
> if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
> minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
> hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
> }
> {code}
> The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-442) Failed to deliver messages due
to inconsistency between SubscriptionState and LedgerRanges.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490182#comment-13490182 ]
Jiannan Wang commented on BOOKKEEPER-442:
-----------------------------------------
Thanks Ivan for review, I'll update the patch.
Sorry for delay reply, I'm make a moving and cannot access internet these days.
> Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
> -------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-442
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-442.diff
>
>
> The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
> The issue is described as below:
> 1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
> {code}
> AbstractSubscriptionManager:
>
> if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
> }
> {code}
> 2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
> {code}
> public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
> long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
> if (interval <= 0) {
> return false;
> }
> // set consume seq id when it is larger
> this.lastConsumeSeqId = lastConsumeSeqId;
> if (interval < consumeInterval) {
> return false;
> }
> // subscription state will be updated, marked it as clean
> subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
> return true;
> }
> {code}
> 3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
> {code}
> for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
> if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
> minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
> hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
> }
> {code}
> The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-442) Failed to deliver messages due
to inconsistency between SubscriptionState and LedgerRanges.
Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504707#comment-13504707 ]
Ivan Kelly commented on BOOKKEEPER-442:
---------------------------------------
I added a new comment to the review board. Basically, it's not enough to just log either, as that will hang the client request. You need to complete the callback also.
> Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
> -------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-442
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-442.diff, BOOKKEEPER-442.diff
>
>
> The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
> The issue is described as below:
> 1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
> {code}
> AbstractSubscriptionManager:
>
> if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
> }
> {code}
> 2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
> {code}
> public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
> long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
> if (interval <= 0) {
> return false;
> }
> // set consume seq id when it is larger
> this.lastConsumeSeqId = lastConsumeSeqId;
> if (interval < consumeInterval) {
> return false;
> }
> // subscription state will be updated, marked it as clean
> subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
> return true;
> }
> {code}
> 3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
> {code}
> for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
> if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
> minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
> hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
> }
> {code}
> The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-442) Failed to deliver messages due to
inconsistency between SubscriptionState and LedgerRanges.
Posted by "Jiannan Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiannan Wang updated BOOKKEEPER-442:
------------------------------------
Attachment: BOOKKEEPER-442.diff
Update patch by Ivan's suggestion, also update it in review board.
> Failed to deliver messages due to inconsistency between SubscriptionState and LedgerRanges.
> -------------------------------------------------------------------------------------------
>
> Key: BOOKKEEPER-442
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-442
> Project: Bookkeeper
> Issue Type: Bug
> Components: hedwig-server
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Sijie Guo
> Assignee: Jiannan Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: BOOKKEEPER-442.diff, BOOKKEEPER-442.diff, BOOKKEEPER-442.diff
>
>
> The problems encountered when failed to updateSubscriptionState but deleted consumed ledgers.
> The issue is described as below:
> 1) A subscriber setLastConsumeSeqId to move consume ptr. If the consume ptr is moved over consume interval, an update subscription state operation is issued to update to ZooKeeper.
> {code}
> AbstractSubscriptionManager:
>
> if (subState.setLastConsumeSeqId(consumeSeqId, cfg.getConsumeInterval())) { updateSubscriptionState(topic, subscriberId, subState, cb, ctx);
> }
> {code}
> 2) when move consume ptr, it also changed in-memory subscription state before the subscription state is persisted to ZooKeeper.
> {code}
> public boolean setLastConsumeSeqId(MessageSeqId lastConsumeSeqId, int consumeInterval) {
> long interval = lastConsumeSeqId.getLocalComponent() - subscriptionState.getMsgId(). getLocalComponent();
> if (interval <= 0) {
> return false;
> }
> // set consume seq id when it is larger
> this.lastConsumeSeqId = lastConsumeSeqId;
> if (interval < consumeInterval) {
> return false;
> }
> // subscription state will be updated, marked it as clean
> subscriptionState = SubscriptionState.newBuilder(subscriptionState). setMsgId(lastConsumeSeqId).build();
> return true;
> }
> {code}
> 3) MessageConsumedTask runs periodically to delete consumed ledgers. it would use in-memory subscription state to perform such deletion. so if ledger is deleted first and failed to update subscription state. it would cause inconsistent state, when hub restarts and subscriber reconnects, it would use old seq id to start delivering but the ledger has messages with old seq id has been deleted.
> {code}
> for (InMemorySubscriptionState curSubscription : topicSubscriptions.values()) {
> if (curSubscription.getSubscriptionState().getMsgId().getLocalComponent() < minConsumedMessage)
> minConsumedMessage = curSubscription.getSubscriptionState().getMsgId(). getLocalComponent();
> hasBound = hasBound && curSubscription.getSubscriptionPreferences(). hasMessageBound();
> }
> {code}
> The fix would be let message consume task only use persistence state to performance deletions only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira