You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Mark Gellings (JIRA)" <ji...@apache.org> on 2010/03/15 18:33:45 UTC
[jira] Created: (AMQNET-243) failover causes duplicate messages
failover causes duplicate messages
----------------------------------
Key: AMQNET-243
URL: https://issues.apache.org/activemq/browse/AMQNET-243
Project: ActiveMQ .Net
Issue Type: Bug
Components: NMS
Affects Versions: 1.1.0
Reporter: Mark Gellings
Assignee: Jim Gomes
Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
Attached to that issue is a zip file which is password protected with password "fridaytest".
We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
To replicate:
1) Work through the console prompts and produce 50 msgs.
2) Restart console and start consuming those 50 msgs.
3) In the middle of the consumer processing, restart broker
4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
5) Then NMS blows up which seems like a different problem?
>From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
[1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
Regards,
Mark
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (AMQNET-243) failover causes duplicate
messages
Posted by "Mark Gellings (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58840#action_58840 ]
Mark Gellings edited comment on AMQNET-243 at 4/14/10 3:22 PM:
---------------------------------------------------------------
Attached is sample code on how we implemented an Idempotent Consumer pattern on top of NMS.
Basic execution logic goes like this...the IdempotentListener<TMessage> Prehook( ) method is called before a message is processed. If it is the original message (redelivered counter 0) then we check to see if the message is in the database, if it is we send it through as redelivered (by throwing our RecoverableMessageException) so the listener logic can process as a redelivery.
Notes:
--When coding this I tried to follow the Idempotent Consumer pattern in Apache Camel as close as possible
--I understand some of this code may not make sense (e.g. extension methods, it's a layer of abstraction above ActiveMQ.NMS, some code is missing). Unfortunately I can't submit that code.
--While it may make sense to have a Camel.Net sub project, it may be overkill as I don't see other EIP being implemented. If adding this logic to NMS, the database connection needs to remain agnostic. As you can see we used iBATIS.NET to connect to the database, so theoretically any database platform could be used. But ultimately we wouldn't want ActiveMQ.NMS to have iBATIS dependencies. Perhaps an IMessageDao implementation would be required and set on initialization to enable an Idempotent Consumer. Then NMS is not responsible for the implementation of the database access.
All else aside, attached is an example of how to code it.
was (Author: magellings):
Attached is sample code on how we implemented an Idempotent Consumer pattern on top of NMS.
Basic execution logic goes like this...the IdempotentListener<TMessage> Prehook( ) method is called before a message is processed. If it is the original message (redelivered counter 0) then we check to see if the message is in the database, if it is we send it through as redelivered (by throwing our RecoverableMessageException) so the listener logic can process as a redelivery.
Notes:
--When coding this I tried to follow the Idempotent Consumer pattern in Apache Camel.
--I understand some of this code may not make sense (e.g. extension methods, it's a layer of abstraction above ActiveMQ.NMS, some code is missing). Unfortunately I can't submit that code.
--While it may make sense to have a Camel.Net sub project, it may be overkill as I don't see other EIP being implemented. If adding this logic to NMS, the database connection needs to remain agnostic. As you can see we used iBATIS.NET to connect to the database, so theoretically any database platform could be used. But ultimately we wouldn't want ActiveMQ.NMS to have iBATIS dependencies. Perhaps an IMessageDao implementation would be required and set on initialization to enable an Idempotent Consumer. Then NMS is not responsible for the implementation of the database access.
All else aside, attached is an example of how to code it.
> failover causes duplicate messages
> ----------------------------------
>
> Key: AMQNET-243
> URL: https://issues.apache.org/activemq/browse/AMQNET-243
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.1.0
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Attachments: IdempotentConsumer.zip
>
>
> Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
> Attached to that issue is a zip file which is password protected with password "fridaytest".
> We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
> To replicate:
> 1) Work through the console prompts and produce 50 msgs.
> 2) Restart console and start consuming those 50 msgs.
> 3) In the middle of the consumer processing, restart broker
> 4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
> 5) Then NMS blows up which seems like a different problem?
> From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
> Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
> [1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
> Regards,
> Mark
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (AMQNET-243) failover causes duplicate
messages
Posted by "Mark Gellings (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58840#action_58840 ]
Mark Gellings edited comment on AMQNET-243 at 4/14/10 3:23 PM:
---------------------------------------------------------------
Attached is sample code on how we implemented an Idempotent Consumer pattern on top of NMS.
Basic execution logic goes like this...the IdempotentListener<TMessage> Prehook( ) method is called before a message is processed. If it is the original message (redelivered counter 0) then we check to see if the message is in the database, if it is we send it through as redelivered (by throwing our RecoverableMessageException) so the listener logic can process as a redelivery.
Notes:
--When coding this I tried to follow the Idempotent Consumer pattern in Apache Camel as close as possible
--I understand some of this code may not make sense (e.g. extension methods, it's a layer of abstraction above ActiveMQ.NMS, some code is missing). Unfortunately I can't submit that code.
--While it may make sense to have a Camel.Net sub project, it may be overkill as I don't see other EIP patterns being implemented. If adding this logic to NMS, the database connection needs to remain agnostic. As you can see we used iBATIS.NET to connect to the database, so theoretically any database platform could be used. But ultimately we wouldn't want ActiveMQ.NMS to have iBATIS dependencies. Perhaps an IMessageDao implementation would be required and set on initialization to enable an Idempotent Consumer. Then NMS is not responsible for the implementation of the database access.
All else aside, attached is an example of how to code it.
was (Author: magellings):
Attached is sample code on how we implemented an Idempotent Consumer pattern on top of NMS.
Basic execution logic goes like this...the IdempotentListener<TMessage> Prehook( ) method is called before a message is processed. If it is the original message (redelivered counter 0) then we check to see if the message is in the database, if it is we send it through as redelivered (by throwing our RecoverableMessageException) so the listener logic can process as a redelivery.
Notes:
--When coding this I tried to follow the Idempotent Consumer pattern in Apache Camel as close as possible
--I understand some of this code may not make sense (e.g. extension methods, it's a layer of abstraction above ActiveMQ.NMS, some code is missing). Unfortunately I can't submit that code.
--While it may make sense to have a Camel.Net sub project, it may be overkill as I don't see other EIP being implemented. If adding this logic to NMS, the database connection needs to remain agnostic. As you can see we used iBATIS.NET to connect to the database, so theoretically any database platform could be used. But ultimately we wouldn't want ActiveMQ.NMS to have iBATIS dependencies. Perhaps an IMessageDao implementation would be required and set on initialization to enable an Idempotent Consumer. Then NMS is not responsible for the implementation of the database access.
All else aside, attached is an example of how to code it.
> failover causes duplicate messages
> ----------------------------------
>
> Key: AMQNET-243
> URL: https://issues.apache.org/activemq/browse/AMQNET-243
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.1.0
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Attachments: IdempotentConsumer.zip
>
>
> Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
> Attached to that issue is a zip file which is password protected with password "fridaytest".
> We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
> To replicate:
> 1) Work through the console prompts and produce 50 msgs.
> 2) Restart console and start consuming those 50 msgs.
> 3) In the middle of the consumer processing, restart broker
> 4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
> 5) Then NMS blows up which seems like a different problem?
> From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
> Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
> [1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
> Regards,
> Mark
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQNET-243) failover causes duplicate messages
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58275#action_58275 ]
Timothy Bish commented on AMQNET-243:
-------------------------------------
I'd recommend you take a look at the Java client code. There we have some auditing built into the framework to suppress duplicate messages that prevents this from happening as often, there's no such support built into the NMS framework currently.
> failover causes duplicate messages
> ----------------------------------
>
> Key: AMQNET-243
> URL: https://issues.apache.org/activemq/browse/AMQNET-243
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.1.0
> Reporter: Mark Gellings
> Assignee: Jim Gomes
>
> Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
> Attached to that issue is a zip file which is password protected with password "fridaytest".
> We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
> To replicate:
> 1) Work through the console prompts and produce 50 msgs.
> 2) Restart console and start consuming those 50 msgs.
> 3) In the middle of the consumer processing, restart broker
> 4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
> 5) Then NMS blows up which seems like a different problem?
> From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
> Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
> [1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
> Regards,
> Mark
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQNET-243) failover causes duplicate messages
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58250#action_58250 ]
Timothy Bish commented on AMQNET-243:
-------------------------------------
I'd recommend testing this against the official 1.2.0 release as that's been out for quite some time now.
> failover causes duplicate messages
> ----------------------------------
>
> Key: AMQNET-243
> URL: https://issues.apache.org/activemq/browse/AMQNET-243
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.1.0
> Reporter: Mark Gellings
> Assignee: Jim Gomes
>
> Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
> Attached to that issue is a zip file which is password protected with password "fridaytest".
> We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
> To replicate:
> 1) Work through the console prompts and produce 50 msgs.
> 2) Restart console and start consuming those 50 msgs.
> 3) In the middle of the consumer processing, restart broker
> 4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
> 5) Then NMS blows up which seems like a different problem?
> From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
> Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
> [1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
> Regards,
> Mark
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQNET-243) failover causes duplicate messages
Posted by "Mark Gellings (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58274#action_58274 ]
Mark Gellings commented on AMQNET-243:
--------------------------------------
Same problem with official 1.2.0.
We are going to start coding a fix for this at our company within the next few days. Preferably we'd fix in NMS to benefit everyone, but we'd need some guidance on this.
Otherwise we'll just build it into our framework which sits on top of NMS.
> failover causes duplicate messages
> ----------------------------------
>
> Key: AMQNET-243
> URL: https://issues.apache.org/activemq/browse/AMQNET-243
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.1.0
> Reporter: Mark Gellings
> Assignee: Jim Gomes
>
> Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
> Attached to that issue is a zip file which is password protected with password "fridaytest".
> We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
> To replicate:
> 1) Work through the console prompts and produce 50 msgs.
> 2) Restart console and start consuming those 50 msgs.
> 3) In the middle of the consumer processing, restart broker
> 4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
> 5) Then NMS blows up which seems like a different problem?
> From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
> Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
> [1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
> Regards,
> Mark
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQNET-243) failover causes duplicate messages
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58841#action_58841 ]
Timothy Bish commented on AMQNET-243:
-------------------------------------
I'm on travel this week, will take a look next week.
> failover causes duplicate messages
> ----------------------------------
>
> Key: AMQNET-243
> URL: https://issues.apache.org/activemq/browse/AMQNET-243
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.1.0
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Attachments: IdempotentConsumer.zip
>
>
> Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
> Attached to that issue is a zip file which is password protected with password "fridaytest".
> We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
> To replicate:
> 1) Work through the console prompts and produce 50 msgs.
> 2) Restart console and start consuming those 50 msgs.
> 3) In the middle of the consumer processing, restart broker
> 4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
> 5) Then NMS blows up which seems like a different problem?
> From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
> Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
> [1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
> Regards,
> Mark
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (AMQNET-243) failover causes duplicate messages
Posted by "Mark Gellings (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=58276#action_58276 ]
Mark Gellings commented on AMQNET-243:
--------------------------------------
Problem is we need it to not happen at all. When processing idempotent messages they cannot be processed twice. So I don't think duplicating the java code would fix this.
> failover causes duplicate messages
> ----------------------------------
>
> Key: AMQNET-243
> URL: https://issues.apache.org/activemq/browse/AMQNET-243
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.1.0
> Reporter: Mark Gellings
> Assignee: Jim Gomes
>
> Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
> Attached to that issue is a zip file which is password protected with password "fridaytest".
> We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
> To replicate:
> 1) Work through the console prompts and produce 50 msgs.
> 2) Restart console and start consuming those 50 msgs.
> 3) In the middle of the consumer processing, restart broker
> 4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
> 5) Then NMS blows up which seems like a different problem?
> From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
> Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
> [1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
> Regards,
> Mark
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQNET-243) failover causes duplicate messages
Posted by "Mark Gellings (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Gellings updated AMQNET-243:
---------------------------------
Attachment: IdempotentConsumer.zip
Attached is sample code on how we implemented an Idempotent Consumer pattern on top of NMS.
Basic execution logic goes like this...the IdempotentListener<TMessage> Prehook( ) method is called before a message is processed. If it is the original message (redelivered counter 0) then we check to see if the message is in the database, if it is we send it through as redelivered (by throwing our RecoverableMessageException) so the listener logic can process as a redelivery.
Notes:
--When coding this I tried to follow the Idempotent Consumer pattern in Apache Camel.
--I understand some of this code may not make sense (e.g. extension methods, it's a layer of abstraction above ActiveMQ.NMS, some code is missing). Unfortunately I can't submit that code.
--While it may make sense to have a Camel.Net sub project, it may be overkill as I don't see other EIP being implemented. If adding this logic to NMS, the database connection needs to remain agnostic. As you can see we used iBATIS.NET to connect to the database, so theoretically any database platform could be used. But ultimately we wouldn't want ActiveMQ.NMS to have iBATIS dependencies. Perhaps an IMessageDao implementation would be required and set on initialization to enable an Idempotent Consumer. Then NMS is not responsible for the implementation of the database access.
All else aside, attached is an example of how to code it.
> failover causes duplicate messages
> ----------------------------------
>
> Key: AMQNET-243
> URL: https://issues.apache.org/activemq/browse/AMQNET-243
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.1.0
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Attachments: IdempotentConsumer.zip
>
>
> Reference https://issues.apache.org/activemq/browse/AMQ-2627 for specifics on problem.
> Attached to that issue is a zip file which is password protected with password "fridaytest".
> We're using ActiveMQ v5.2, jdbc master/slave MSSQL 2008. Attached is an NMS v1.2 RC4 consumer with a transacted session as well as activemq.xml.
> To replicate:
> 1) Work through the console prompts and produce 50 msgs.
> 2) Restart console and start consuming those 50 msgs.
> 3) In the middle of the consumer processing, restart broker
> 4) The last message consumer was processing will be resent and not marked as redelivered. (this is the idempotent msg problem. Ex. - instead of $500 getting deposited into your account, $1000 does)
> 5) Then NMS blows up which seems like a different problem?
> From what I understand this shouldn't be the case if you use a transacted session, however the attached console app can prove it is a problem.
> Bottomline--I thought this was why the camel idempotent consumer pattern [1] existed which can be leveraged by java clients.
> [1] http://fusesource.com/docs/router/1.6/eip/MsgEnd-Idempotent.html
> Regards,
> Mark
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira