You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "Gunawan, Rahman (GSFC-703.H)[Halvik Corp]" <ra...@nasa.gov.INVALID> on 2022/06/01 18:15:24 UTC
ActiveMQ 5.15.4 duplicate message detection
Hi,
Does ActiveMQ 5.15.4 have duplicate message detection? If yes, where can I find the documentation?
Thanks
Regards,
Rahman
Re: [EXTERNAL] Re: ActiveMQ 5.15.4 duplicate message detection
Posted by Simon Lundström <si...@su.se.INVALID>.
Another way is just not to care about dupes from producers and instead
focus on the consumer via the idempotent receiver enterprise integration
pattern[1] since you probably want do that anyway if you want to be sure
not to process the same message multiple times.
As long as the producer uses a stable and unique ID for every message
that is idempotent it won't matter.
Pretty much what Justin said in his mail and also
https://stackoverflow.com/q/62506865 and in the Artemis docs[2].
BR,
- Simon
1,
<https://www.enterpriseintegrationpatterns.com/patterns/messaging/IdempotentReceiver.html>
2, <https://activemq.apache.org/components/artemis/documentation/latest/duplicate-detection.html>
On Wed, 2022-06-01 at 21:01:19 +0200, Justin Bertram wrote:
> To my knowledge, ActiveMQ "Classic" doesn't support detecting duplicate
> messages from producers. You might consider using ActiveMQ Artemis if you
> want that feature.
>
>
> Justin
>
> On Wed, Jun 1, 2022 at 1:35 PM Gunawan, Rahman (GSFC-703.H)[Halvik Corp]
> <ra...@nasa.gov.invalid> wrote:
>
> > The broker detecting duplicates from producers.
> > Thanks
> >
> > Rahman
> >
> > -----Original Message-----
> > From: Justin Bertram <jb...@apache.org>
> > Sent: Wednesday, June 1, 2022 2:30 PM
> > To: users@activemq.apache.org
> > Subject: [EXTERNAL] Re: ActiveMQ 5.15.4 duplicate message detection
> >
> > What's the use-case? Are you talking about the broker detecting duplicates
> > from producers or consumers detecting duplicates from the broker?
> >
> >
> > Justin
> >
> > On Wed, Jun 1, 2022 at 1:16 PM Gunawan, Rahman (GSFC-703.H)[Halvik Corp] <
> > rahman.gunawan@nasa.gov.invalid> wrote:
> >
> > > Hi,
> > > Does ActiveMQ 5.15.4 have duplicate message detection? If yes, where
> > > can I find the documentation?
> > >
> > > Thanks
> > >
> > > Regards,
> > > Rahman
> > >
> >
Re: [EXTERNAL] Re: ActiveMQ 5.15.4 duplicate message detection
Posted by Justin Bertram <jb...@apache.org>.
To my knowledge, ActiveMQ "Classic" doesn't support detecting duplicate
messages from producers. You might consider using ActiveMQ Artemis if you
want that feature.
Justin
On Wed, Jun 1, 2022 at 1:35 PM Gunawan, Rahman (GSFC-703.H)[Halvik Corp]
<ra...@nasa.gov.invalid> wrote:
> The broker detecting duplicates from producers.
> Thanks
>
> Rahman
>
> -----Original Message-----
> From: Justin Bertram <jb...@apache.org>
> Sent: Wednesday, June 1, 2022 2:30 PM
> To: users@activemq.apache.org
> Subject: [EXTERNAL] Re: ActiveMQ 5.15.4 duplicate message detection
>
> What's the use-case? Are you talking about the broker detecting duplicates
> from producers or consumers detecting duplicates from the broker?
>
>
> Justin
>
> On Wed, Jun 1, 2022 at 1:16 PM Gunawan, Rahman (GSFC-703.H)[Halvik Corp] <
> rahman.gunawan@nasa.gov.invalid> wrote:
>
> > Hi,
> > Does ActiveMQ 5.15.4 have duplicate message detection? If yes, where
> > can I find the documentation?
> >
> > Thanks
> >
> > Regards,
> > Rahman
> >
>
RE: [EXTERNAL] Re: ActiveMQ 5.15.4 duplicate message detection
Posted by "Gunawan, Rahman (GSFC-703.H)[Halvik Corp]" <ra...@nasa.gov.INVALID>.
The broker detecting duplicates from producers.
Thanks
Rahman
-----Original Message-----
From: Justin Bertram <jb...@apache.org>
Sent: Wednesday, June 1, 2022 2:30 PM
To: users@activemq.apache.org
Subject: [EXTERNAL] Re: ActiveMQ 5.15.4 duplicate message detection
What's the use-case? Are you talking about the broker detecting duplicates from producers or consumers detecting duplicates from the broker?
Justin
On Wed, Jun 1, 2022 at 1:16 PM Gunawan, Rahman (GSFC-703.H)[Halvik Corp] <ra...@nasa.gov.invalid> wrote:
> Hi,
> Does ActiveMQ 5.15.4 have duplicate message detection? If yes, where
> can I find the documentation?
>
> Thanks
>
> Regards,
> Rahman
>
Re: ActiveMQ 5.15.4 duplicate message detection
Posted by Justin Bertram <jb...@apache.org>.
What's the use-case? Are you talking about the broker detecting duplicates
from producers or consumers detecting duplicates from the broker?
Justin
On Wed, Jun 1, 2022 at 1:16 PM Gunawan, Rahman (GSFC-703.H)[Halvik Corp]
<ra...@nasa.gov.invalid> wrote:
> Hi,
> Does ActiveMQ 5.15.4 have duplicate message detection? If yes, where can
> I find the documentation?
>
> Thanks
>
> Regards,
> Rahman
>
Re: ActiveMQ 5.15.4 duplicate message detection
Posted by Matt Pavlovich <ma...@gmail.com>.
Hi Rahman-
ActiveMQ 5.x includes duplicate message detection in both client and broker-side areas. For failover producers, there is the ability to track messages and take steps to flush or reduce duplicate during failover.
The broker contains audit points at several places (Destinations and KahaDB store) to look for message dupes (based on message ID) that accounts for several of the low-level and broker-to-broker scenarios. Additionally, when messages are paged out, dupe checking is available to make sure duplicate messages are not paged-in from the store (recent fix added a counter metric for this, instead of copying duplicate detected messages to the DLQ: https://issues.apache.org/jira/browse/AMQ-8397)
[Client Connection Factory]
setAuditDepth
ref: https://activemq.apache.org/maven/apidocs/org/apache/activemq/ActiveMQConnectionFactory.html
[Failover Transport]
trackMessages
ref: https://activemq.apache.org/failover-transport-reference.html
[Destination Policies]
enableAudit = true | false
maxAuditDepth
maxProducersToAudit
ref: https://activemq.apache.org/per-destination-policies
[KahaDB]
maxFailoverProducersToTrack
failoverProducersAuditDepth
Are you able to share more specifics about your use case or messaging pattern? ‘Duplicate messages’ can be a tricky thing in distributed messaging (regardless of product used) — sometimes it is a resend of the same message (ie failover), sometimes it is a ‘redelivery’ to the consumer (ie. consumer rollback or does not ack properly). In clusters, you can have redundant publishing of messages (ie. Pub-sub storm) to account for.
Thanks,
Matt Pavlovich
> On Jun 1, 2022, at 1:15 PM, Gunawan, Rahman (GSFC-703.H)[Halvik Corp] <ra...@nasa.gov.INVALID> wrote:
>
> Hi,
> Does ActiveMQ 5.15.4 have duplicate message detection? If yes, where can I find the documentation?
>
> Thanks
>
> Regards,
> Rahman