You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Agalatea <va...@gmail.com> on 2018/04/19 10:40:31 UTC
ActiveMQ.DLQ filled up with TopicSubdiscard and Duplicated
deliveries
I'm running ActiveMQ 5.15.0.
My ActiveMQ.DLQ is filling up with messages with dlqDeliveryFailureCause
like following:
java.lang.Throwable: TopicSubDiscard.
ID:ID:babbbbc63a56-37427-1523628495562-1:44:1:1
java.lang.Throwable: Suppressing duplicate delivery on connection, consumer
ID:82e8ed600671-38460-1517479481261-1:2:1:1
And I'm hitting storage limit too often.
How should I clean up ActiveMQ.DLQ in automatic way so that they expire or
what to do to not have them generated at all?
These messages with dlqDeliveryFailureCause different than expiration of
message don't respect policies - they are not given expiration time set in
sharedDeadLetterStrategy.
My policies are following:
<policyEntry topic=">">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy
limit="1000" />
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue=">">
<deadLetterStrategy>
<sharedDeadLetterStrategy expiration="86400000"
/>
</deadLetterStrategy>
</policyEntry>
<policyEntry queue="ABC.Q.ABCINT.>"
prioritizedMessages="true" expireMessagesPeriod="1000">
<deadLetterStrategy>
<sharedDeadLetterStrategy
processNonPersistent="true" expiration="86400000">
<deadLetterQueue>
<queue physicalName="ABC.Q.ABCINT.DLQ"
/>
</deadLetterQueue>
</sharedDeadLetterStrategy>
</deadLetterStrategy>
</policyEntry>
So far I tried:
<timeStampingBrokerPlugin zeroExpirationOverride="86400000" />
but expiration is not set for these "TopicSubDiscard" and "Suppressing
duplicate delivery" messages.
I also tried:
<discardingDLQBrokerPlugin dropOnly="ActiveMQ.DLQ" reportInterval="1000"/>
But then my policies for setting ABC.Q.ABCINT.DLQ as DLQ of ABC.Q.ABCINT.>
stopped working, discarding plugin overrides them.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ.DLQ filled up with TopicSubdiscard and Duplicated deliveries
Posted by Anna Jaworska <va...@gmail.com>.
About:
<discardingDLQBrokerPlugin dropOnly="ActiveMQ.DLQ" reportInterval="1000"/>
But then my policies for setting ABC.Q.ABCINT.DLQ as DLQ of ABC.Q.ABCINT.>
stopped working, discarding plugin overrides them.
After debugging I found that discarding plugin has dropAll behavior always
on, so without explicitly setting dropAll to false i.e.:
<discardingDLQBrokerPlugin dropOnly="ActiveMQ.DLQ" dropAll="false"
reportInterval="1000"/>
it works as if drop all was enabled and that overwrites custom policies.
Also these messages with dlqDeliveryFailureCause :
java.lang.Throwable: TopicSubDiscard.
ID:ID:babbbbc63a56-37427-1523628495562-1:44:1:1
java.lang.Throwable: Suppressing duplicate delivery on connection, consumer
ID:82e8ed600671-38460-1517479481261-1:2:1:1
don't seem to go through sendToDeadLetterQueue() .
I ended up so far with my own plugin that is overwriting
public void send(ProducerBrokerExchange producerExchange, Message messageSend)
that is dropping all messages that enter ActiveMQ.DLQ.
Still any hints what produces "TopicSubDiscard" and "Suppressing duplicate
delivery on connection" are welcome. I presume these are produced by end
application not broker itself.
2018-04-24 7:34 GMT+02:00 Tim Bain <tb...@alumni.duke.edu>:
> I'm confused about why you're seeing the behavior you're seeing.
>
> Looking at
> org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(), I
> can see that we get the dead letter strategy for the topic, then copy the
> message and use the dead letter strategy's expiration value to set the
> message's expiration value. I don't see anything obvious that would explain
> that value not being set on the message when it's moved to the DLQ. Would
> you be able to get the ActiveMQ source for the version you're using, and
> then set a breakpoint in that method to see why that code isn't doing
> what's expected based on code inspection?
>
> Also, it looks like you're not following the warning in the middle of
> http://activemq.apache.org/message-redelivery-and-dlq-handling.html, since
> your queue="ABC.Q.ABCINT.>" policyEntry matches your DLQ name. I'm not sure
> that this would cause the behavior you're seeing, but it certainly doesn't
> look right, so you should fix it as part of these changes.
>
> Finally, looking at the source for the DiscardingDLQBrokerPlugin, I'm not
> seeing anything that would change the behavior of which DLQ to use for a
> given topic. It looks to me like
> DiscardingDLQBroker.sendToDeadLetterQueue() should just call
> next.sendToDeadLetterQueue(), which would apply the same dead letter
> strategy. Can you share any more information about this, or step through
> that code with a debugger?
>
> Tim
>
> On Thu, Apr 19, 2018 at 4:40 AM, Agalatea <va...@gmail.com> wrote:
>
> > I'm running ActiveMQ 5.15.0.
> >
> > My ActiveMQ.DLQ is filling up with messages with dlqDeliveryFailureCause
> > like following:
> > java.lang.Throwable: TopicSubDiscard.
> > ID:ID:babbbbc63a56-37427-1523628495562-1:44:1:1
> > java.lang.Throwable: Suppressing duplicate delivery on connection,
> consumer
> > ID:82e8ed600671-38460-1517479481261-1:2:1:1
> >
> > And I'm hitting storage limit too often.
> >
> > How should I clean up ActiveMQ.DLQ in automatic way so that they expire
> or
> > what to do to not have them generated at all?
> >
> > These messages with dlqDeliveryFailureCause different than expiration of
> > message don't respect policies - they are not given expiration time set
> in
> > sharedDeadLetterStrategy.
> >
> > My policies are following:
> >
> > <policyEntry topic=">">
> > <pendingMessageLimitStrategy>
> > <constantPendingMessageLimitStrategy
> > limit="1000" />
> > </pendingMessageLimitStrategy>
> > </policyEntry>
> > <policyEntry queue=">">
> > <deadLetterStrategy>
> > <sharedDeadLetterStrategy
> expiration="86400000"
> > />
> > </deadLetterStrategy>
> > </policyEntry>
> >
> > <policyEntry queue="ABC.Q.ABCINT.>"
> > prioritizedMessages="true" expireMessagesPeriod="1000">
> > <deadLetterStrategy>
> > <sharedDeadLetterStrategy
> > processNonPersistent="true" expiration="86400000">
> > <deadLetterQueue>
> > <queue physicalName="ABC.Q.ABCINT.
> DLQ"
> > />
> > </deadLetterQueue>
> > </sharedDeadLetterStrategy>
> > </deadLetterStrategy>
> > </policyEntry>
> >
> >
> >
> > So far I tried:
> > <timeStampingBrokerPlugin zeroExpirationOverride="86400000" />
> > but expiration is not set for these "TopicSubDiscard" and "Suppressing
> > duplicate delivery" messages.
> >
> >
> > I also tried:
> > <discardingDLQBrokerPlugin dropOnly="ActiveMQ.DLQ"
> reportInterval="1000"/>
> > But then my policies for setting ABC.Q.ABCINT.DLQ as DLQ of
> ABC.Q.ABCINT.>
> > stopped working, discarding plugin overrides them.
> >
> >
> >
> >
> >
> >
> >
> > --
> > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> > f2341805.html
> >
>
Re: ActiveMQ.DLQ filled up with TopicSubdiscard and Duplicated deliveries
Posted by Tim Bain <tb...@alumni.duke.edu>.
I'm confused about why you're seeing the behavior you're seeing.
Looking at
org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(), I
can see that we get the dead letter strategy for the topic, then copy the
message and use the dead letter strategy's expiration value to set the
message's expiration value. I don't see anything obvious that would explain
that value not being set on the message when it's moved to the DLQ. Would
you be able to get the ActiveMQ source for the version you're using, and
then set a breakpoint in that method to see why that code isn't doing
what's expected based on code inspection?
Also, it looks like you're not following the warning in the middle of
http://activemq.apache.org/message-redelivery-and-dlq-handling.html, since
your queue="ABC.Q.ABCINT.>" policyEntry matches your DLQ name. I'm not sure
that this would cause the behavior you're seeing, but it certainly doesn't
look right, so you should fix it as part of these changes.
Finally, looking at the source for the DiscardingDLQBrokerPlugin, I'm not
seeing anything that would change the behavior of which DLQ to use for a
given topic. It looks to me like
DiscardingDLQBroker.sendToDeadLetterQueue() should just call
next.sendToDeadLetterQueue(), which would apply the same dead letter
strategy. Can you share any more information about this, or step through
that code with a debugger?
Tim
On Thu, Apr 19, 2018 at 4:40 AM, Agalatea <va...@gmail.com> wrote:
> I'm running ActiveMQ 5.15.0.
>
> My ActiveMQ.DLQ is filling up with messages with dlqDeliveryFailureCause
> like following:
> java.lang.Throwable: TopicSubDiscard.
> ID:ID:babbbbc63a56-37427-1523628495562-1:44:1:1
> java.lang.Throwable: Suppressing duplicate delivery on connection, consumer
> ID:82e8ed600671-38460-1517479481261-1:2:1:1
>
> And I'm hitting storage limit too often.
>
> How should I clean up ActiveMQ.DLQ in automatic way so that they expire or
> what to do to not have them generated at all?
>
> These messages with dlqDeliveryFailureCause different than expiration of
> message don't respect policies - they are not given expiration time set in
> sharedDeadLetterStrategy.
>
> My policies are following:
>
> <policyEntry topic=">">
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy
> limit="1000" />
> </pendingMessageLimitStrategy>
> </policyEntry>
> <policyEntry queue=">">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy expiration="86400000"
> />
> </deadLetterStrategy>
> </policyEntry>
>
> <policyEntry queue="ABC.Q.ABCINT.>"
> prioritizedMessages="true" expireMessagesPeriod="1000">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy
> processNonPersistent="true" expiration="86400000">
> <deadLetterQueue>
> <queue physicalName="ABC.Q.ABCINT.DLQ"
> />
> </deadLetterQueue>
> </sharedDeadLetterStrategy>
> </deadLetterStrategy>
> </policyEntry>
>
>
>
> So far I tried:
> <timeStampingBrokerPlugin zeroExpirationOverride="86400000" />
> but expiration is not set for these "TopicSubDiscard" and "Suppressing
> duplicate delivery" messages.
>
>
> I also tried:
> <discardingDLQBrokerPlugin dropOnly="ActiveMQ.DLQ" reportInterval="1000"/>
> But then my policies for setting ABC.Q.ABCINT.DLQ as DLQ of ABC.Q.ABCINT.>
> stopped working, discarding plugin overrides them.
>
>
>
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>