You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Brad Harvey <Br...@gbst.com> on 2022/02/16 08:07:59 UTC
Artemis duplicate detection
Hi,
I'll put my questions up the top so they don't get lost, see background below for the context.
Question 1 - I do not want duplicate detection (1) to roll back entire transactions. Is there a way to change the behaviour so that the entire transaction does not get rolled back? The behaviour I would like is to either silently drop the single duplicate message and commit the rest of the transaction, or not be enabled at all (as the application has duplicate detection).
Question 2 - I think consuming a message from artemis using the openwire protocol sets the _AMQ_DUPL_ID header on the consumed message which is available to the client to read (2). Is there anything in the activemq classic openwire or qpid jms amqp libraries that would result in this property being set under the covers when sending a message?
Background:
I ran into issue where artemis' duplicate detection triggered, causing transaction rollback. I say unexpected because we don't set the duplicate detection header explicitly in our code.
Artemis 2.20.0. Communication is using the Openwire protocol with ActiveMQ Classic JMS client libraries.
The application is Apache Camel based. There's a route that reads multiple messages from queue "A", does "stuff", then sends them onto another queue "B" using a JMS transacted session.
The problem encountered was a failure in committing the transaction after writing to queue "B":
2022-02-14 12:52:16.898 INFO ||ID-syn-ang-rbau-csp-1a-1644803524964-0-40| 1593489 --- [n.FromCSPSigned] org.apache.activemq.TransactionContext : commit failed for transaction TX:ID:hostname-44825-1644803526098-1:1:4
javax.jms.JMSException: org.apache.activemq.artemis.api.core.ActiveMQDuplicateIdException: Duplicate message detected - message will not be routed. Message information:CoreMessage[ <snip> _AMQ_DUPL_ID=ID:hostname1-42351-1643245903980-1:132:2746:5:1, <snip>
This caused a rollback & retry in the camel route (expected) which continued to fail with the same exception until retries were exceeded and the batch of messages was moved from queue "A" to the DLQ.
I believe the way the header got onto the message being sent was:
1. When sending messages to an openwire consumer that uses the failover transport, artemis adds the _AMQ_DUPL_ID header to the message. (2)
2. Camel copies all JMS headers to camel message headers when consuming a JMS message.
3. Camel copies all camel messages headers to JMS headers when sending a JMS message.
And so #3 dutifully set the _AMQ_DUPL_ID header. I think the solution is for our camel application to use header filters on the camel components so that #2/#3 does not occur. But I'm concerned about other ways this header might get set - hence question 1 and 2 at the start of the email.
Thanks for your time and guidance.
Brad Harvey
(1) https://activemq.apache.org/components/artemis/documentation/latest/duplicate-detection.html
(2) https://github.com/apache/activemq-artemis/blob/1e6297957766138c312273cbbd15327c22f91c53/artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/amq/AMQSession.java#L397
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.
RE: Artemis duplicate detection
Posted by Brad Harvey <Br...@gbst.com>.
No problems, Gary, and I appreciate your input.
I've created
https://issues.apache.org/jira/browse/ARTEMIS-3693
-----Original Message-----
From: Gary Tully <ga...@gmail.com>
Sent: Thursday, 17 February 2022 9:53 PM
To: users@activemq.apache.org
Subject: Re: Artemis duplicate detection
apologies, I must have been dreaming of how it should be! my bad. I know there were multiple cases where it needed to be disabled for protocols.
the only other thing that is left is to set the duplicate id cache size to 1, it does not support any value <= 0.
that is broker wide however, so not configurable at the address level, it can maybe work ok for transaction batches > 1, but an immediate resend of the same message will be rejected.
id-cache-size = 1 in broker.xml as referenced by
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fduplicate-detection.html&data=04%7C01%7CBrad.Harvey%40gbst.com%7C75d95f7ad3394440777b08d9f20c0c7a%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0%7C637806955849663699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pyW2v8wEhkA6VU%2BZk4S4hyBiY6DzkEfAJI4lmWk1HaY%3D&reserved=0
Would you mind raising a JIRA issue with your use case and wish as to how it should be configured? Really just capturing this email thread.
On Thu, 17 Feb 2022 at 09:59, Brad Harvey <Br...@gbst.com> wrote:
>
> Hi Gary,
>
> Thanks for the tips. Turning off the duplicate detection at the address level sounds like what I'm after, but I couldn't spot an applicable setting at the address level - only in the bridge/cluster connections.
>
> I looked in
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Facti
> vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fdupl
> icate-detection.html&data=04%7C01%7CBrad.Harvey%40gbst.com%7C75d95
> f7ad3394440777b08d9f20c0c7a%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0
> %7C637806955849663699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pyW2v8wEhk
> A6VU%2BZk4S4hyBiY6DzkEfAJI4lmWk1HaY%3D&reserved=0
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Facti
> vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fconf
> iguration-index.html&data=04%7C01%7CBrad.Harvey%40gbst.com%7C75d95
> f7ad3394440777b08d9f20c0c7a%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0
> %7C637806955849663699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pzoTS7kNhi
> 63ruTNLQ8Q9wMRUH3Zt2iguKpSW8TyhgQ%3D&reserved=0
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fapache%2Factivemq-artemis%2Fblob%2Fmain%2Fartemis-server%2Fsr
> c%2Fmain%2Fresources%2Fschema%2Fartemis-configuration.xsd&data=04%
> 7C01%7CBrad.Harvey%40gbst.com%7C75d95f7ad3394440777b08d9f20c0c7a%7C1c2
> da354196b481891e4f760cbaac9e4%7C0%7C0%7C637806955849663699%7CUnknown%7
> CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXV
> CI6Mn0%3D%7C3000&sdata=W2R%2B76d8hU9DXKUuQS02Ek1Ar31lhImbjtZ9v5lLQ
> Ko%3D&reserved=0
>
> Thanks
> Brad
>
> -----Original Message-----
> From: Gary Tully <ga...@gmail.com>
> Sent: Wednesday, 16 February 2022 8:45 PM
> To: users@activemq.apache.org
> Subject: Re: Artemis duplicate detection
>
> The fix in https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-3521&data=04%7C01%7CBrad.Harvey%40gbst.com%7C75d95f7ad3394440777b08d9f20c0c7a%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0%7C637806955849663699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=e82%2FjCdu6zn9eBqZTA%2FapCYpoJvBXI5GKFjT40pCEMM%3D&reserved=0 gives you the option to disable the use of duplicate id by the openwire protocol adapter.
> see example usage in the test:
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitb
> ox.apache.org%2Frepos%2Fasf%3Fp%3Dactivemq-artemis.git%3Ba%3Dblob%3Bf%
> 3Dtests%2Fintegration-tests%2Fsrc%2Ftest%2Fjava%2Forg%2Fapache%2Factiv
> emq%2Fartemis%2Ftests%2Fintegration%2Fopenwire%2FFailoverDuplicateIDUs
> ageTest.java%3Bh%3D3231256e1a5d037d364844221813ef51392eb02e%3Bhb%3D15c
> 2dbb%23l54&data=04%7C01%7CBrad.Harvey%40gbst.com%7C75d95f7ad339444
> 0777b08d9f20c0c7a%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0%7C6378069
> 55849663699%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMz
> IiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dbHqoOZ6MjSxKPijwBd4
> m9rPVz96Q36Mdbnsq08Eb0M%3D&reserved=0
>
> Otherwise, the use of duplicate detection is configurable on an address level via address settings, you could disable it altogether with a wildcard match.
>
> On Wed, 16 Feb 2022 at 08:08, Brad Harvey <Br...@gbst.com> wrote:
> >
> > Hi,
> >
> > I'll put my questions up the top so they don't get lost, see background below for the context.
> >
> > Question 1 - I do not want duplicate detection (1) to roll back entire transactions. Is there a way to change the behaviour so that the entire transaction does not get rolled back? The behaviour I would like is to either silently drop the single duplicate message and commit the rest of the transaction, or not be enabled at all (as the application has duplicate detection).
> >
> > Question 2 - I think consuming a message from artemis using the openwire protocol sets the _AMQ_DUPL_ID header on the consumed message which is available to the client to read (2). Is there anything in the activemq classic openwire or qpid jms amqp libraries that would result in this property being set under the covers when sending a message?
> >
> >
> > Background:
> >
> > I ran into issue where artemis' duplicate detection triggered, causing transaction rollback. I say unexpected because we don't set the duplicate detection header explicitly in our code.
> >
> > Artemis 2.20.0. Communication is using the Openwire protocol with ActiveMQ Classic JMS client libraries.
> > The application is Apache Camel based. There's a route that reads multiple messages from queue "A", does "stuff", then sends them onto another queue "B" using a JMS transacted session.
> >
> > The problem encountered was a failure in committing the transaction after writing to queue "B":
> >
> > 2022-02-14 12:52:16.898 INFO ||ID-syn-ang-rbau-csp-1a-1644803524964-0-40| 1593489 --- [n.FromCSPSigned] org.apache.activemq.TransactionContext : commit failed for transaction TX:ID:hostname-44825-1644803526098-1:1:4
> > javax.jms.JMSException: org.apache.activemq.artemis.api.core.ActiveMQDuplicateIdException: Duplicate message detected - message will not be routed. Message information:CoreMessage[ <snip> _AMQ_DUPL_ID=ID:hostname1-42351-1643245903980-1:132:2746:5:1, <snip>
> >
> > This caused a rollback & retry in the camel route (expected) which continued to fail with the same exception until retries were exceeded and the batch of messages was moved from queue "A" to the DLQ.
> >
> > I believe the way the header got onto the message being sent was:
> > 1. When sending messages to an openwire consumer that uses the
> > failover transport, artemis adds the _AMQ_DUPL_ID header to the message. (2) 2. Camel copies all JMS headers to camel message headers when consuming a JMS message.
> > 3. Camel copies all camel messages headers to JMS headers when sending a JMS message.
> >
> > And so #3 dutifully set the _AMQ_DUPL_ID header. I think the solution is for our camel application to use header filters on the camel components so that #2/#3 does not occur. But I'm concerned about other ways this header might get set - hence question 1 and 2 at the start of the email.
> >
> > Thanks for your time and guidance.
> > Brad Harvey
> >
> > (1)
> > https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fac
> > ti
> > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fdu
> > pl
> > icate-detection.html&data=04%7C01%7CBrad.Harvey%40gbst.com%7Cead
> > 9b
> > fed28a7438968c508d9f1397115%7C1c2da354196b481891e4f760cbaac9e4%7C0%7
> > C0
> > %7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC
> > JQ
> > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=UQd3Bfe5
> > Jl
> > 3GNmZQXGy%2BRTCL54WY829sBeqpdOM%2BGZA%3D&reserved=0
> > (2)
> > https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > th
> > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2F1e6297957766138c312273cb
> > bd
> > 15327c22f91c53%2Fartemis-protocols%2Fartemis-openwire-protocol%2Fsrc
> > %2
> > Fmain%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Fcore%2Fprotocol%2
> > Fo
> > penwire%2Famq%2FAMQSession.java%23L397&data=04%7C01%7CBrad.Harve
> > y%
> > 40gbst.com%7Cead9bfed28a7438968c508d9f1397115%7C1c2da354196b481891e4
> > f7
> > 60cbaac9e4%7C0%7C0%7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJW
> > Ij
> > oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&
> > am
> > p;sdata=RZLPGmOSk%2BGAL5AGjs2wH39vz%2B%2Bg5XQc5bzzzJf5dpo%3D&res
> > er
> > ved=0
> >
> > The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.
Re: Artemis duplicate detection
Posted by Gary Tully <ga...@gmail.com>.
apologies, I must have been dreaming of how it should be! my bad. I
know there were multiple cases where it needed to be disabled for
protocols.
the only other thing that is left is to set the duplicate id cache
size to 1, it does not support any value <= 0.
that is broker wide however, so not configurable at the address level,
it can maybe work ok for transaction batches > 1, but an immediate
resend of the same message will be rejected.
id-cache-size = 1 in broker.xml as referenced by
https://activemq.apache.org/components/artemis/documentation/latest/duplicate-detection.html
Would you mind raising a JIRA issue with your use case and wish as to
how it should be configured? Really just capturing this email thread.
On Thu, 17 Feb 2022 at 09:59, Brad Harvey <Br...@gbst.com> wrote:
>
> Hi Gary,
>
> Thanks for the tips. Turning off the duplicate detection at the address level sounds like what I'm after, but I couldn't spot an applicable setting at the address level - only in the bridge/cluster connections.
>
> I looked in
> https://activemq.apache.org/components/artemis/documentation/latest/duplicate-detection.html
> https://activemq.apache.org/components/artemis/documentation/latest/configuration-index.html
> https://github.com/apache/activemq-artemis/blob/main/artemis-server/src/main/resources/schema/artemis-configuration.xsd
>
> Thanks
> Brad
>
> -----Original Message-----
> From: Gary Tully <ga...@gmail.com>
> Sent: Wednesday, 16 February 2022 8:45 PM
> To: users@activemq.apache.org
> Subject: Re: Artemis duplicate detection
>
> The fix in https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-3521&data=04%7C01%7CBrad.Harvey%40gbst.com%7Cead9bfed28a7438968c508d9f1397115%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0%7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pmjvly5Eq9gVN6lOCMfTASo8eSj%2BVGERsJJyOWU4cYs%3D&reserved=0 gives you the option to disable the use of duplicate id by the openwire protocol adapter.
> see example usage in the test:
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%3Fp%3Dactivemq-artemis.git%3Ba%3Dblob%3Bf%3Dtests%2Fintegration-tests%2Fsrc%2Ftest%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Ftests%2Fintegration%2Fopenwire%2FFailoverDuplicateIDUsageTest.java%3Bh%3D3231256e1a5d037d364844221813ef51392eb02e%3Bhb%3D15c2dbb%23l54&data=04%7C01%7CBrad.Harvey%40gbst.com%7Cead9bfed28a7438968c508d9f1397115%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0%7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=J9c%2BNsAlFyW4xgMtPKUYnjEg1hSg%2BqOeOQuEvf%2BbJcA%3D&reserved=0
>
> Otherwise, the use of duplicate detection is configurable on an address level via address settings, you could disable it altogether with a wildcard match.
>
> On Wed, 16 Feb 2022 at 08:08, Brad Harvey <Br...@gbst.com> wrote:
> >
> > Hi,
> >
> > I'll put my questions up the top so they don't get lost, see background below for the context.
> >
> > Question 1 - I do not want duplicate detection (1) to roll back entire transactions. Is there a way to change the behaviour so that the entire transaction does not get rolled back? The behaviour I would like is to either silently drop the single duplicate message and commit the rest of the transaction, or not be enabled at all (as the application has duplicate detection).
> >
> > Question 2 - I think consuming a message from artemis using the openwire protocol sets the _AMQ_DUPL_ID header on the consumed message which is available to the client to read (2). Is there anything in the activemq classic openwire or qpid jms amqp libraries that would result in this property being set under the covers when sending a message?
> >
> >
> > Background:
> >
> > I ran into issue where artemis' duplicate detection triggered, causing transaction rollback. I say unexpected because we don't set the duplicate detection header explicitly in our code.
> >
> > Artemis 2.20.0. Communication is using the Openwire protocol with ActiveMQ Classic JMS client libraries.
> > The application is Apache Camel based. There's a route that reads multiple messages from queue "A", does "stuff", then sends them onto another queue "B" using a JMS transacted session.
> >
> > The problem encountered was a failure in committing the transaction after writing to queue "B":
> >
> > 2022-02-14 12:52:16.898 INFO ||ID-syn-ang-rbau-csp-1a-1644803524964-0-40| 1593489 --- [n.FromCSPSigned] org.apache.activemq.TransactionContext : commit failed for transaction TX:ID:hostname-44825-1644803526098-1:1:4
> > javax.jms.JMSException: org.apache.activemq.artemis.api.core.ActiveMQDuplicateIdException: Duplicate message detected - message will not be routed. Message information:CoreMessage[ <snip> _AMQ_DUPL_ID=ID:hostname1-42351-1643245903980-1:132:2746:5:1, <snip>
> >
> > This caused a rollback & retry in the camel route (expected) which continued to fail with the same exception until retries were exceeded and the batch of messages was moved from queue "A" to the DLQ.
> >
> > I believe the way the header got onto the message being sent was:
> > 1. When sending messages to an openwire consumer that uses the
> > failover transport, artemis adds the _AMQ_DUPL_ID header to the message. (2) 2. Camel copies all JMS headers to camel message headers when consuming a JMS message.
> > 3. Camel copies all camel messages headers to JMS headers when sending a JMS message.
> >
> > And so #3 dutifully set the _AMQ_DUPL_ID header. I think the solution is for our camel application to use header filters on the camel components so that #2/#3 does not occur. But I'm concerned about other ways this header might get set - hence question 1 and 2 at the start of the email.
> >
> > Thanks for your time and guidance.
> > Brad Harvey
> >
> > (1)
> > https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Facti
> > vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fdupl
> > icate-detection.html&data=04%7C01%7CBrad.Harvey%40gbst.com%7Cead9b
> > fed28a7438968c508d9f1397115%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0
> > %7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=UQd3Bfe5Jl
> > 3GNmZQXGy%2BRTCL54WY829sBeqpdOM%2BGZA%3D&reserved=0
> > (2)
> > https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> > ub.com%2Fapache%2Factivemq-artemis%2Fblob%2F1e6297957766138c312273cbbd
> > 15327c22f91c53%2Fartemis-protocols%2Fartemis-openwire-protocol%2Fsrc%2
> > Fmain%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Fcore%2Fprotocol%2Fo
> > penwire%2Famq%2FAMQSession.java%23L397&data=04%7C01%7CBrad.Harvey%
> > 40gbst.com%7Cead9bfed28a7438968c508d9f1397115%7C1c2da354196b481891e4f7
> > 60cbaac9e4%7C0%7C0%7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> > oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&am
> > p;sdata=RZLPGmOSk%2BGAL5AGjs2wH39vz%2B%2Bg5XQc5bzzzJf5dpo%3D&reser
> > ved=0
> >
> > The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.
RE: Artemis duplicate detection
Posted by Brad Harvey <Br...@gbst.com>.
Hi Gary,
Thanks for the tips. Turning off the duplicate detection at the address level sounds like what I'm after, but I couldn't spot an applicable setting at the address level - only in the bridge/cluster connections.
I looked in
https://activemq.apache.org/components/artemis/documentation/latest/duplicate-detection.html
https://activemq.apache.org/components/artemis/documentation/latest/configuration-index.html
https://github.com/apache/activemq-artemis/blob/main/artemis-server/src/main/resources/schema/artemis-configuration.xsd
Thanks
Brad
-----Original Message-----
From: Gary Tully <ga...@gmail.com>
Sent: Wednesday, 16 February 2022 8:45 PM
To: users@activemq.apache.org
Subject: Re: Artemis duplicate detection
The fix in https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARTEMIS-3521&data=04%7C01%7CBrad.Harvey%40gbst.com%7Cead9bfed28a7438968c508d9f1397115%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0%7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pmjvly5Eq9gVN6lOCMfTASo8eSj%2BVGERsJJyOWU4cYs%3D&reserved=0 gives you the option to disable the use of duplicate id by the openwire protocol adapter.
see example usage in the test:
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitbox.apache.org%2Frepos%2Fasf%3Fp%3Dactivemq-artemis.git%3Ba%3Dblob%3Bf%3Dtests%2Fintegration-tests%2Fsrc%2Ftest%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Ftests%2Fintegration%2Fopenwire%2FFailoverDuplicateIDUsageTest.java%3Bh%3D3231256e1a5d037d364844221813ef51392eb02e%3Bhb%3D15c2dbb%23l54&data=04%7C01%7CBrad.Harvey%40gbst.com%7Cead9bfed28a7438968c508d9f1397115%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0%7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=J9c%2BNsAlFyW4xgMtPKUYnjEg1hSg%2BqOeOQuEvf%2BbJcA%3D&reserved=0
Otherwise, the use of duplicate detection is configurable on an address level via address settings, you could disable it altogether with a wildcard match.
On Wed, 16 Feb 2022 at 08:08, Brad Harvey <Br...@gbst.com> wrote:
>
> Hi,
>
> I'll put my questions up the top so they don't get lost, see background below for the context.
>
> Question 1 - I do not want duplicate detection (1) to roll back entire transactions. Is there a way to change the behaviour so that the entire transaction does not get rolled back? The behaviour I would like is to either silently drop the single duplicate message and commit the rest of the transaction, or not be enabled at all (as the application has duplicate detection).
>
> Question 2 - I think consuming a message from artemis using the openwire protocol sets the _AMQ_DUPL_ID header on the consumed message which is available to the client to read (2). Is there anything in the activemq classic openwire or qpid jms amqp libraries that would result in this property being set under the covers when sending a message?
>
>
> Background:
>
> I ran into issue where artemis' duplicate detection triggered, causing transaction rollback. I say unexpected because we don't set the duplicate detection header explicitly in our code.
>
> Artemis 2.20.0. Communication is using the Openwire protocol with ActiveMQ Classic JMS client libraries.
> The application is Apache Camel based. There's a route that reads multiple messages from queue "A", does "stuff", then sends them onto another queue "B" using a JMS transacted session.
>
> The problem encountered was a failure in committing the transaction after writing to queue "B":
>
> 2022-02-14 12:52:16.898 INFO ||ID-syn-ang-rbau-csp-1a-1644803524964-0-40| 1593489 --- [n.FromCSPSigned] org.apache.activemq.TransactionContext : commit failed for transaction TX:ID:hostname-44825-1644803526098-1:1:4
> javax.jms.JMSException: org.apache.activemq.artemis.api.core.ActiveMQDuplicateIdException: Duplicate message detected - message will not be routed. Message information:CoreMessage[ <snip> _AMQ_DUPL_ID=ID:hostname1-42351-1643245903980-1:132:2746:5:1, <snip>
>
> This caused a rollback & retry in the camel route (expected) which continued to fail with the same exception until retries were exceeded and the batch of messages was moved from queue "A" to the DLQ.
>
> I believe the way the header got onto the message being sent was:
> 1. When sending messages to an openwire consumer that uses the
> failover transport, artemis adds the _AMQ_DUPL_ID header to the message. (2) 2. Camel copies all JMS headers to camel message headers when consuming a JMS message.
> 3. Camel copies all camel messages headers to JMS headers when sending a JMS message.
>
> And so #3 dutifully set the _AMQ_DUPL_ID header. I think the solution is for our camel application to use header filters on the camel components so that #2/#3 does not occur. But I'm concerned about other ways this header might get set - hence question 1 and 2 at the start of the email.
>
> Thanks for your time and guidance.
> Brad Harvey
>
> (1)
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Facti
> vemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fdupl
> icate-detection.html&data=04%7C01%7CBrad.Harvey%40gbst.com%7Cead9b
> fed28a7438968c508d9f1397115%7C1c2da354196b481891e4f760cbaac9e4%7C0%7C0
> %7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=UQd3Bfe5Jl
> 3GNmZQXGy%2BRTCL54WY829sBeqpdOM%2BGZA%3D&reserved=0
> (2)
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fapache%2Factivemq-artemis%2Fblob%2F1e6297957766138c312273cbbd
> 15327c22f91c53%2Fartemis-protocols%2Fartemis-openwire-protocol%2Fsrc%2
> Fmain%2Fjava%2Forg%2Fapache%2Factivemq%2Fartemis%2Fcore%2Fprotocol%2Fo
> penwire%2Famq%2FAMQSession.java%23L397&data=04%7C01%7CBrad.Harvey%
> 40gbst.com%7Cead9bfed28a7438968c508d9f1397115%7C1c2da354196b481891e4f7
> 60cbaac9e4%7C0%7C0%7C637806051335437239%7CUnknown%7CTWFpbGZsb3d8eyJWIj
> oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&am
> p;sdata=RZLPGmOSk%2BGAL5AGjs2wH39vz%2B%2Bg5XQc5bzzzJf5dpo%3D&reser
> ved=0
>
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.
Re: Artemis duplicate detection
Posted by Gary Tully <ga...@gmail.com>.
The fix in https://issues.apache.org/jira/browse/ARTEMIS-3521 gives
you the option to disable the use of duplicate id by the openwire
protocol adapter.
see example usage in the test:
https://gitbox.apache.org/repos/asf?p=activemq-artemis.git;a=blob;f=tests/integration-tests/src/test/java/org/apache/activemq/artemis/tests/integration/openwire/FailoverDuplicateIDUsageTest.java;h=3231256e1a5d037d364844221813ef51392eb02e;hb=15c2dbb#l54
Otherwise, the use of duplicate detection is configurable on an
address level via address settings, you could disable it altogether
with a wildcard match.
On Wed, 16 Feb 2022 at 08:08, Brad Harvey <Br...@gbst.com> wrote:
>
> Hi,
>
> I'll put my questions up the top so they don't get lost, see background below for the context.
>
> Question 1 - I do not want duplicate detection (1) to roll back entire transactions. Is there a way to change the behaviour so that the entire transaction does not get rolled back? The behaviour I would like is to either silently drop the single duplicate message and commit the rest of the transaction, or not be enabled at all (as the application has duplicate detection).
>
> Question 2 - I think consuming a message from artemis using the openwire protocol sets the _AMQ_DUPL_ID header on the consumed message which is available to the client to read (2). Is there anything in the activemq classic openwire or qpid jms amqp libraries that would result in this property being set under the covers when sending a message?
>
>
> Background:
>
> I ran into issue where artemis' duplicate detection triggered, causing transaction rollback. I say unexpected because we don't set the duplicate detection header explicitly in our code.
>
> Artemis 2.20.0. Communication is using the Openwire protocol with ActiveMQ Classic JMS client libraries.
> The application is Apache Camel based. There's a route that reads multiple messages from queue "A", does "stuff", then sends them onto another queue "B" using a JMS transacted session.
>
> The problem encountered was a failure in committing the transaction after writing to queue "B":
>
> 2022-02-14 12:52:16.898 INFO ||ID-syn-ang-rbau-csp-1a-1644803524964-0-40| 1593489 --- [n.FromCSPSigned] org.apache.activemq.TransactionContext : commit failed for transaction TX:ID:hostname-44825-1644803526098-1:1:4
> javax.jms.JMSException: org.apache.activemq.artemis.api.core.ActiveMQDuplicateIdException: Duplicate message detected - message will not be routed. Message information:CoreMessage[ <snip> _AMQ_DUPL_ID=ID:hostname1-42351-1643245903980-1:132:2746:5:1, <snip>
>
> This caused a rollback & retry in the camel route (expected) which continued to fail with the same exception until retries were exceeded and the batch of messages was moved from queue "A" to the DLQ.
>
> I believe the way the header got onto the message being sent was:
> 1. When sending messages to an openwire consumer that uses the failover transport, artemis adds the _AMQ_DUPL_ID header to the message. (2)
> 2. Camel copies all JMS headers to camel message headers when consuming a JMS message.
> 3. Camel copies all camel messages headers to JMS headers when sending a JMS message.
>
> And so #3 dutifully set the _AMQ_DUPL_ID header. I think the solution is for our camel application to use header filters on the camel components so that #2/#3 does not occur. But I'm concerned about other ways this header might get set - hence question 1 and 2 at the start of the email.
>
> Thanks for your time and guidance.
> Brad Harvey
>
> (1) https://activemq.apache.org/components/artemis/documentation/latest/duplicate-detection.html
> (2) https://github.com/apache/activemq-artemis/blob/1e6297957766138c312273cbbd15327c22f91c53/artemis-protocols/artemis-openwire-protocol/src/main/java/org/apache/activemq/artemis/core/protocol/openwire/amq/AMQSession.java#L397
>
> The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and / or privileged material that may be governed by confidential information provisions contained in the agreement between GBST and your company. Any disclosure, copying, distribution, or other use without the express consent of the sender is prohibited. If you received this in error, please contact the sender and delete the material from any computer. All rights in the information transmitted, including copyright, are reserved. Nothing in this message should be interpreted as a digital signature that can be used to authenticate a document. No warranty is given by the sender that any attachments to this email are free from viruses or other defects.