You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Martin Ritchie (JIRA)" <qp...@incubator.apache.org> on 2009/04/15 19:54:15 UTC
[jira] Created: (QPID-1816) Unable to acknowledge messages after
failover
Unable to acknowledge messages after failover
---------------------------------------------
Key: QPID-1816
URL: https://issues.apache.org/jira/browse/QPID-1816
Project: Qpid
Issue Type: Bug
Components: Java Client
Reporter: Martin Ritchie
Priority: Blocker
Fix For: 0.5
Summary:
If messages have been consumed during a session but not acknowledged and failover occurs it is not possible to acknowledge subsequent messages.
In the transacted case calling rollback after failover clears the 'dirty' state of the connection.
However calling recover does not do the same work so the dirty session remains and users are unable to acknowledge messages.
This occurred on a CLIENT_ACK session but I have not yet investigated the other Acking modes so they may be unaffected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Issue Comment Edited: (QPID-1816) Unable to acknowledge
messages after failover
Posted by "Martin Ritchie (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707870#action_12707870 ]
Martin Ritchie edited comment on QPID-1816 at 9/22/09 5:41 AM:
---------------------------------------------------------------
This is too big a change too late for 0.5
was (Author: ritchiem):
This is too big a change to late for 0.5
> Unable to acknowledge messages after failover
> ---------------------------------------------
>
> Key: QPID-1816
> URL: https://issues.apache.org/jira/browse/QPID-1816
> Project: Qpid
> Issue Type: Bug
> Components: Java Client
> Reporter: Martin Ritchie
> Assignee: Martin Ritchie
> Priority: Critical
>
> Summary:
> If messages have been consumed during a session but not acknowledged and failover occurs it is not possible to acknowledge subsequent messages.
> In the transacted case calling rollback after failover clears the 'dirty' state of the connection.
> However calling recover does not do the same work so the dirty session remains and users are unable to acknowledge messages.
> This occurred on a CLIENT_ACK session but I have not yet investigated the other Acking modes so they may be unaffected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Updated: (QPID-1816) Unable to acknowledge messages after
failover
Posted by "Martin Ritchie (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Ritchie updated QPID-1816:
---------------------------------
Priority: Critical (was: Blocker)
Fix Version/s: (was: 0.5)
This is too big a change to late for 0.5
> Unable to acknowledge messages after failover
> ---------------------------------------------
>
> Key: QPID-1816
> URL: https://issues.apache.org/jira/browse/QPID-1816
> Project: Qpid
> Issue Type: Bug
> Components: Java Client
> Reporter: Martin Ritchie
> Priority: Critical
>
> Summary:
> If messages have been consumed during a session but not acknowledged and failover occurs it is not possible to acknowledge subsequent messages.
> In the transacted case calling rollback after failover clears the 'dirty' state of the connection.
> However calling recover does not do the same work so the dirty session remains and users are unable to acknowledge messages.
> This occurred on a CLIENT_ACK session but I have not yet investigated the other Acking modes so they may be unaffected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Assigned: (QPID-1816) Unable to acknowledge messages after
failover
Posted by "Martin Ritchie (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Martin Ritchie reassigned QPID-1816:
------------------------------------
Assignee: Martin Ritchie
> Unable to acknowledge messages after failover
> ---------------------------------------------
>
> Key: QPID-1816
> URL: https://issues.apache.org/jira/browse/QPID-1816
> Project: Qpid
> Issue Type: Bug
> Components: Java Client
> Reporter: Martin Ritchie
> Assignee: Martin Ritchie
> Priority: Critical
>
> Summary:
> If messages have been consumed during a session but not acknowledged and failover occurs it is not possible to acknowledge subsequent messages.
> In the transacted case calling rollback after failover clears the 'dirty' state of the connection.
> However calling recover does not do the same work so the dirty session remains and users are unable to acknowledge messages.
> This occurred on a CLIENT_ACK session but I have not yet investigated the other Acking modes so they may be unaffected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Commented: (QPID-1816) Unable to acknowledge messages after
failover
Posted by "Martin Ritchie (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759590#action_12759590 ]
Martin Ritchie commented on QPID-1816:
--------------------------------------
This trace shows multiple acks are sent but the Java broker ignores the duplicate.
In a real app the first ack would be for a much higher delivery tag and so would fail.
main 2009-09-25 16:31:04,163 DEBUG [qpid.client.AMQSession.Dispatcher] Set Dispatcher Connection Started: Currently Started
main 2009-09-25 16:32:14,501 DEBUG [apache.qpid.client.AMQSession] Sending ack for delivery tag 1 on channel 1
main 2009-09-25 16:32:14,530 DEBUG [apache.qpid.framing.AMQDataBlockEncoder] Encoded frame byte-buffer is '0x0100010000000d003c0050000000000000000100ce'
pool-1-thread-4 2009-09-25 16:32:14,572 DEBUG [qpid.server.protocol.AMQProtocolSession] Frame Received: Frame channelId: 1, bodyFrame: [BasicAckBodyImpl: deliveryTag=1, multiple=false]
pool-1-thread-4 2009-09-25 16:32:14,879 DEBUG [qpid.server.handler.BasicAckMethodHandler] Ack(Tag:1:Mult:false) received on channel 1
pool-1-thread-4 2009-09-25 16:32:14,881 DEBUG [qpid.server.txn.LocalTransactionalContext] Starting transaction on message store: org.apache.qpid.server.txn.LocalTransactionalContext@159d10
main 2009-09-25 16:32:31,661 DEBUG [apache.qpid.client.AMQSession] Sending ack for delivery tag 1 on channel 1
main 2009-09-25 16:32:31,697 DEBUG [apache.qpid.framing.AMQDataBlockEncoder] Encoded frame byte-buffer is '0x0100010000000d003c0050000000000000000100ce'
pool-1-thread-3 2009-09-25 16:32:32,071 DEBUG [qpid.server.protocol.AMQProtocolSession] Frame Received: Frame channelId: 1, bodyFrame: [BasicAckBodyImpl: deliveryTag=1, multiple=false]
pool-1-thread-3 2009-09-25 16:32:32,072 DEBUG [qpid.server.handler.BasicAckMethodHandler] Ack(Tag:1:Mult:false) received on channel 1
main 2009-09-25 16:32:46,049 DEBUG [apache.qpid.framing.AMQDataBlockEncoder] Encoded frame byte-buffer is '0x01000100000004005a0014ce'
pool-1-thread-2 2009-09-25 16:32:46,098 DEBUG [qpid.server.protocol.AMQProtocolSession] Frame Received: Frame channelId: 1, bodyFrame: [TxCommitBodyImpl: ]
pool-1-thread-2 2009-09-25 16:32:46,129 DEBUG [qpid.server.handler.TxCommitHandler] Commit received on channel 1
pool-1-thread-2 2009-09-25 16:32:46,153 DEBUG [qpid.server.txn.LocalTransactionalContext] Committing transactional context: org.apache.qpid.server.txn.LocalTransactionalContext@159d10
pool-1-thread-2 2009-09-25 16:32:46,194 DEBUG [qpid.server.txn.TxnBuffer] Committing 2 ops to commit.:[org.apache.qpid.server.ack.TxAck@f32dde, org.apache.qpid.server.txn.StoreMessageOperation@1487b8b]
pool-1-thread-2 2009-09-25 16:32:46,235 DEBUG [qpid.server.store.MemoryMessageStore] Removing message with id 1
pool-1-thread-2 2009-09-25 16:32:46,262 DEBUG [qpid.server.txn.LocalTransactionalContext] Performing post commit delivery
pool-1-thread-3 2009-09-25 16:32:46,293 DEBUG [apache.qpid.framing.AMQDataBlockEncoder] Encoded frame byte-buffer is '0x01000100000004005a0015ce'
pool-1-thread-3 2009-09-25 16:32:46,327 DEBUG [qpid.server.protocol.AMQPFastProtocolHandler] Message sent: DirectBuffer[pos=0 lim=12 cap=16: 01 00 01 00 00 00 04 00 5A 00 15 CE]
pool-1-thread-3 2009-09-25 16:32:46,364 DEBUG [qpid.server.protocol.AMQPFastProtocolHandler] Message sent: Frame channelId: 1, bodyFrame: [TxCommitOkBodyImpl: ]
pool-1-thread-4 2009-09-25 16:32:46,367 DEBUG [qpid.client.protocol.AMQProtocolHandler] (14133705)Method frame received: [TxCommitOkBodyImpl: ]
> Unable to acknowledge messages after failover
> ---------------------------------------------
>
> Key: QPID-1816
> URL: https://issues.apache.org/jira/browse/QPID-1816
> Project: Qpid
> Issue Type: Bug
> Components: Java Client
> Reporter: Martin Ritchie
> Assignee: Martin Ritchie
> Priority: Critical
>
> Summary:
> If messages have been consumed during a session but not acknowledged and failover occurs it is not possible to acknowledge subsequent messages.
> In the transacted case calling rollback after failover clears the 'dirty' state of the connection.
> However calling recover does not do the same work so the dirty session remains and users are unable to acknowledge messages.
> This occurred on a CLIENT_ACK session but I have not yet investigated the other Acking modes so they may be unaffected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
[jira] Commented: (QPID-1816) Unable to acknowledge messages after
failover
Posted by "Martin Ritchie (JIRA)" <qp...@incubator.apache.org>.
[ https://issues.apache.org/jira/browse/QPID-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759587#action_12759587 ]
Martin Ritchie commented on QPID-1816:
--------------------------------------
I have completed tests that validate the reception of messages after failover. As the JIRA pointed out CLIENT_ACK was broken. The fix for which is very straight forward. Currently in postDeliver the session is marked dirty.. this is wrong.. as it means in onMessage if the message is acknowlege the session is still marked dirty! Move the dirty setting to preDeliver and adding a markClean in acknowledge. Clears this up.
However, After testing receive/onMessage paths with and without failover and seeing no errors. Testing that we correctly throw a 'dirty session' exception from Client_Ack and Transacted sessions when the session is dirty seemed sensible... Problem is only client ack works.
In the transacted case we never mark the session dirty when we receive a message. There for it is possible to receive message 1, failover receive message 2 commit and believe we have commited both messages. Now perhaps my test is a little contrived as in a clustered environment message 1 would be receieved twice unlike my test which is not clustered so preps the new broker with message count... (NUM_MSGS - COUNT).
Whlist the test may not be ideal the fact that we can commit without an exception is wrong as there is no guarrantee that the second message we receive is a redelivery of the first.
As I see it this means that in the event of failover any open transaction whilst it may commit cleanly, what it is committing is not clear.
Testing needs to be done on dirty session due to receive and dirty sessions due to send.
> Unable to acknowledge messages after failover
> ---------------------------------------------
>
> Key: QPID-1816
> URL: https://issues.apache.org/jira/browse/QPID-1816
> Project: Qpid
> Issue Type: Bug
> Components: Java Client
> Reporter: Martin Ritchie
> Assignee: Martin Ritchie
> Priority: Critical
>
> Summary:
> If messages have been consumed during a session but not acknowledged and failover occurs it is not possible to acknowledge subsequent messages.
> In the transacted case calling rollback after failover clears the 'dirty' state of the connection.
> However calling recover does not do the same work so the dirty session remains and users are unable to acknowledge messages.
> This occurred on a CLIENT_ACK session but I have not yet investigated the other Acking modes so they may be unaffected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org