You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Tim Bain <tb...@alumni.duke.edu> on 2017/10/20 05:39:42 UTC
Re: ActiveMQ DLQ issues
This tells you that the messages were discarded by ActiveMQ because they
had expired. Are your producers setting a time to live when sending
messages?
Tim
On Fri, Sep 15, 2017 at 5:42 PM, mtod <mt...@thetods.net> wrote:
> I'm running ActiveMQ 5.10.0 with an embedded camel route. It's been running
> fine until we connected the master to an instance 5.14.5.
>
> 5.10.0 Network Connector:
>
> <networkConnectors>
> <networkConnector name="connector019"
> duplex="true" networkTTL="2"
> userName="${activemq.username}" password="${activemq.password}"
> uri="masterslave:(tcp://XXX01.corp.local:61616,tcp://XXX02.
> corp.local:61616,tcp://XXX03.corp.local:61616)">
> <excludedDestinations>
> <topic
> physicalName="mirth.applogger"/>
> <queue physicalName="esb.>"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <topic physicalName=">" />
> <queue physicalName=">" />
> </staticallyIncludedDestinations>
> </networkConnector>
> </networkConnectors>
>
>
> Now I have been experiencing a lot of messages going to the DLQ.
>
> I looked at the messages and they don't seem to have anything wrong with
> them I did notice the following:
>
> dlqDeliveryFailureCause java.lang.Throwable: TopicSubDiscard.
> ID:localhost->fmiuseastp01-63022-1486246061090-100948:1:1:2
>
> I'm not sure what this is telling me.
>
> Thanks
>
> Mike
>
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: ActiveMQ DLQ issues
Posted by Tim Bain <tb...@alumni.duke.edu>.
Thanks Gary. I was convinced that the default configuration did not contain
any of the message-discarding options from the slow consumers page I linked
to; I'm not sure where I got that idea. But that explanation makes more
sense than my wild theory about the no-consumer behavior of statically
included virtual topics.
Tim
On Nov 20, 2017 4:54 AM, "Gary Tully" <ga...@gmail.com> wrote:
> dlqDeliveryFailureCause java.lang.Throwable: TopicSubDiscard.
> ID:localhost->fmiuseastp01-63022-1486246061090-100948:1:1:2
>
> The cause indicates the a subscriber is full and has been configured to
> discard messages[1]. See policyEntry from the default config [2]
>
> https://github.com/apache/activemq/blob/41a100766c19655816d575841ba559
> d33c63313d/activemq-broker/src/main/java/org/apache/
> activemq/broker/region/TopicSubscription.java#L718
>
> https://github.com/apache/activemq/blob/41a100766c19655816d575841ba559
> d33c63313d/assembly/src/release/conf/activemq.xml#L55
>
> On Mon, 20 Nov 2017 at 11:08 augustl <au...@augustl.com> wrote:
>
> > Thanks for the detailed response!
> >
> > Unfortunately I don't know when the out of memory error happened. My logs
> > were full of them. Messages started appearing
> > in the DLQ about 21:00, and the logs with the OOM start at around 05:00
> the
> > day after.
> >
> > On the producer end, I connect to ActiveMQ using a
> > ActiveMQConnectionFactory
> > with the URL set to `failover:(tcp://localhost:61616)`. I create the
> > connection by calling `createConnection` on this factory. I call
> > `setClientId` on the connection before I call `start` on it to connect.
> To
> > create the session, I call `createSession(false,
> > javax.jms.Session.AUTO_ACKNOWLEDGE)` on the connection object. I then
> call
> > `createProducer` on the session, with the destination being `new
> > ActiveMQTopic("VirtualTopic.MyTopicNameHere")`. The message I send is
> > created by calling `createTextMessage` on the session, and I pass a
> string
> > to it, containing some XML. Then I call `send` on the producer, with the
> > message I just created. This producer uses
> > "org.apache.activemq:activemq-core:5.7.0".
> >
> > The ActiveMQ broker that I connect to runs on version 5.14.4.
> >
> > I don't override any defaults or set any properties on the connection or
> > the
> > session or the producer or the message, so pretty much everything seems
> to
> > be running in default configuration as far as I can tell.
> >
> > The topic I posted messages to, "VirtualTopic.MyTopicNameHere" (which
> isn't
> > the real name, obviously), where the messages ended up in DLQ, is a topic
> > that is automatically created by the producer. I don't have any extra
> > config
> > for it in the activemq config files.
> >
> > Your theory about the consumer for that topic on the remote broker is
> > interesting. Things went a bit haywire so we restarted some things here
> and
> > there before we were able to fully collect information about what
> happened.
> > The broker that we configured as a network connector and that had
> > VirtualTopic.MyTopicNameHere statically included, seemed to have problems
> > as
> > well. I noticed, for example, that this remote broker didn't accept new
> > connections, until we restarted it. So it's possible that the root cause
> is
> > that the remote broker failed, and this caused things to go haywire on
> the
> > local broker.
> >
> > I was a bit surprised that messages to a topic ended up on a DLQ, since
> if
> > nobody consumes a message on a topic, the message is just discarded and
> the
> > world moves on. But you're saying that if a topic is statically included
> as
> > a destination in a network connector, and the broker cannot access that
> > network connector, those messages end up in the DLQ of the local broker?
> > That makes sense to me, at least. In that case, I suppose the only input
> I
> > have here is that it would be nice if the broker that puts the message on
> > the DLQ has a more informative reason in the dlqDeliveryFailureCause
> > attribute :)
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
Re: ActiveMQ DLQ issues
Posted by Gary Tully <ga...@gmail.com>.
dlqDeliveryFailureCause java.lang.Throwable: TopicSubDiscard.
ID:localhost->fmiuseastp01-63022-1486246061090-100948:1:1:2
The cause indicates the a subscriber is full and has been configured to
discard messages[1]. See policyEntry from the default config [2]
https://github.com/apache/activemq/blob/41a100766c19655816d575841ba559d33c63313d/activemq-broker/src/main/java/org/apache/activemq/broker/region/TopicSubscription.java#L718
https://github.com/apache/activemq/blob/41a100766c19655816d575841ba559d33c63313d/assembly/src/release/conf/activemq.xml#L55
On Mon, 20 Nov 2017 at 11:08 augustl <au...@augustl.com> wrote:
> Thanks for the detailed response!
>
> Unfortunately I don't know when the out of memory error happened. My logs
> were full of them. Messages started appearing
> in the DLQ about 21:00, and the logs with the OOM start at around 05:00 the
> day after.
>
> On the producer end, I connect to ActiveMQ using a
> ActiveMQConnectionFactory
> with the URL set to `failover:(tcp://localhost:61616)`. I create the
> connection by calling `createConnection` on this factory. I call
> `setClientId` on the connection before I call `start` on it to connect. To
> create the session, I call `createSession(false,
> javax.jms.Session.AUTO_ACKNOWLEDGE)` on the connection object. I then call
> `createProducer` on the session, with the destination being `new
> ActiveMQTopic("VirtualTopic.MyTopicNameHere")`. The message I send is
> created by calling `createTextMessage` on the session, and I pass a string
> to it, containing some XML. Then I call `send` on the producer, with the
> message I just created. This producer uses
> "org.apache.activemq:activemq-core:5.7.0".
>
> The ActiveMQ broker that I connect to runs on version 5.14.4.
>
> I don't override any defaults or set any properties on the connection or
> the
> session or the producer or the message, so pretty much everything seems to
> be running in default configuration as far as I can tell.
>
> The topic I posted messages to, "VirtualTopic.MyTopicNameHere" (which isn't
> the real name, obviously), where the messages ended up in DLQ, is a topic
> that is automatically created by the producer. I don't have any extra
> config
> for it in the activemq config files.
>
> Your theory about the consumer for that topic on the remote broker is
> interesting. Things went a bit haywire so we restarted some things here and
> there before we were able to fully collect information about what happened.
> The broker that we configured as a network connector and that had
> VirtualTopic.MyTopicNameHere statically included, seemed to have problems
> as
> well. I noticed, for example, that this remote broker didn't accept new
> connections, until we restarted it. So it's possible that the root cause is
> that the remote broker failed, and this caused things to go haywire on the
> local broker.
>
> I was a bit surprised that messages to a topic ended up on a DLQ, since if
> nobody consumes a message on a topic, the message is just discarded and the
> world moves on. But you're saying that if a topic is statically included as
> a destination in a network connector, and the broker cannot access that
> network connector, those messages end up in the DLQ of the local broker?
> That makes sense to me, at least. In that case, I suppose the only input I
> have here is that it would be nice if the broker that puts the message on
> the DLQ has a more informative reason in the dlqDeliveryFailureCause
> attribute :)
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: ActiveMQ DLQ issues
Posted by augustl <au...@augustl.com>.
Thanks for the detailed response!
Unfortunately I don't know when the out of memory error happened. My logs
were full of them. Messages started appearing
in the DLQ about 21:00, and the logs with the OOM start at around 05:00 the
day after.
On the producer end, I connect to ActiveMQ using a ActiveMQConnectionFactory
with the URL set to `failover:(tcp://localhost:61616)`. I create the
connection by calling `createConnection` on this factory. I call
`setClientId` on the connection before I call `start` on it to connect. To
create the session, I call `createSession(false,
javax.jms.Session.AUTO_ACKNOWLEDGE)` on the connection object. I then call
`createProducer` on the session, with the destination being `new
ActiveMQTopic("VirtualTopic.MyTopicNameHere")`. The message I send is
created by calling `createTextMessage` on the session, and I pass a string
to it, containing some XML. Then I call `send` on the producer, with the
message I just created. This producer uses
"org.apache.activemq:activemq-core:5.7.0".
The ActiveMQ broker that I connect to runs on version 5.14.4.
I don't override any defaults or set any properties on the connection or the
session or the producer or the message, so pretty much everything seems to
be running in default configuration as far as I can tell.
The topic I posted messages to, "VirtualTopic.MyTopicNameHere" (which isn't
the real name, obviously), where the messages ended up in DLQ, is a topic
that is automatically created by the producer. I don't have any extra config
for it in the activemq config files.
Your theory about the consumer for that topic on the remote broker is
interesting. Things went a bit haywire so we restarted some things here and
there before we were able to fully collect information about what happened.
The broker that we configured as a network connector and that had
VirtualTopic.MyTopicNameHere statically included, seemed to have problems as
well. I noticed, for example, that this remote broker didn't accept new
connections, until we restarted it. So it's possible that the root cause is
that the remote broker failed, and this caused things to go haywire on the
local broker.
I was a bit surprised that messages to a topic ended up on a DLQ, since if
nobody consumes a message on a topic, the message is just discarded and the
world moves on. But you're saying that if a topic is statically included as
a destination in a network connector, and the broker cannot access that
network connector, those messages end up in the DLQ of the local broker?
That makes sense to me, at least. In that case, I suppose the only input I
have here is that it would be nice if the broker that puts the message on
the DLQ has a more informative reason in the dlqDeliveryFailureCause
attribute :)
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ DLQ issues
Posted by augustl <au...@augustl.com>.
Thanks for the detailed response!
Unfortunately I don't know when the out of memory error happened. My logs
were full of them. Messages started appearing
in the DLQ about 21:00, and the logs with the OOM start at around 05:00 the
day after.
On the producer end, I connect to ActiveMQ using a ActiveMQConnectionFactory
with the URL set to `failover:(tcp://localhost:61616)`. I create the
connection by calling `createConnection` on this factory. I call
`setClientId` on the connection before I call `start` on it to connect. To
create the session, I call `createSession(false,
javax.jms.Session.AUTO_ACKNOWLEDGE)` on the connection object. I then call
`createProducer` on the session, with the destination being `new
ActiveMQTopic("VirtualTopic.MyTopicNameHere")`. The message I send is
created by calling `createTextMessage` on the session, and I pass a string
to it, containing some XML. Then I call `send` on the producer, with the
message I just created. This producer uses
"org.apache.activemq:activemq-core:5.7.0".
The ActiveMQ broker that I connect to runs on version 5.14.4.
I don't override any defaults or set any properties on the connection or the
session or the producer or the message, so pretty much everything seems to
be running in default configuration as far as I can tell.
The topic I posted messages to, "VirtualTopic.MyTopicNameHere" (which isn't
the real name, obviously), where the messages ended up in DLQ, is a topic
that is automatically created by the producer. I don't have any extra config
for it in the activemq config files.
Your theory about the consumer for that topic on the remote broker is
interesting. Things went a bit haywire so we restarted some things here and
there before we were able to fully collect information about what happened.
The broker that we configured as a network connector and that had
VirtualTopic.MyTopicNameHere statically included, seemed to have problems as
well. I noticed, for example, that this remote broker didn't accept new
connections, until we restarted it. So it's possible that the root cause is
that the remote broker failed, and this caused things to go haywire on the
local broker.
I was a bit surprised that messages to a topic ended up on a DLQ, since if
nobody consumes a message on a topic, the message is just discarded and the
world moves on. But you're saying that if a topic is statically included as
a destination in a network connector, and the broker cannot access that
network connector, those messages end up in the DLQ of the local broker?
That makes sense to me, at least. In that case, I suppose the only input I
have here is that it would be nice if the broker that puts the message on
the DLQ has a more informative reason in the dlqDeliveryFailureCause
attribute :)
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ DLQ issues
Posted by augustl <au...@augustl.com>.
Thanks for the detailed response!
Unfortunately I don't know when the out of memory error happened. My logs
were full of them. Messages started appearing
in the DLQ about 21:00, and the logs with the OOM start at around 05:00 the
day after.
On the producer end, I connect to ActiveMQ using a ActiveMQConnectionFactory
with the URL set to `failover:(tcp://localhost:61616)`. I create the
connection by calling `createConnection` on this factory. I call
`setClientId` on the connection before I call `start` on it to connect. To
create the session, I call `createSession(false,
javax.jms.Session.AUTO_ACKNOWLEDGE)` on the connection object. I then call
`createProducer` on the session, with the destination being `new
ActiveMQTopic("VirtualTopic.MyTopicNameHere")`. The message I send is
created by calling `createTextMessage` on the session, and I pass a string
to it, containing some XML. Then I call `send` on the producer, with the
message I just created. This producer uses
"org.apache.activemq:activemq-core:5.7.0".
The ActiveMQ broker that I connect to runs on version 5.14.4.
I don't override any defaults or set any properties on the connection or the
session or the producer or the message, so pretty much everything seems to
be running in default configuration as far as I can tell.
The topic I posted messages to, "VirtualTopic.MyTopicNameHere" (which isn't
the real name, obviously), where the messages ended up in DLQ, is a topic
that is automatically created by the producer. I don't have any extra config
for it in the activemq config files.
Your theory about the consumer for that topic on the remote broker is
interesting. Things went a bit haywire so we restarted some things here and
there before we were able to fully collect information about what happened.
The broker that we configured as a network connector and that had
VirtualTopic.MyTopicNameHere statically included, seemed to have problems as
well. I noticed, for example, that this remote broker didn't accept new
connections, until we restarted it. So it's possible that the root cause is
that the remote broker failed, and this caused things to go haywire on the
local broker.
I was a bit surprised that messages to a topic ended up on a DLQ, since if
nobody consumes a message on a topic, the message is just discarded and the
world moves on. But you're saying that if a topic is statically included as
a destination in a network connector, and the broker cannot access that
network connector, those messages end up in the DLQ of the local broker?
That makes sense to me, at least. In that case, I suppose the only input I
have here is that it would be nice if the broker that puts the message on
the DLQ has a more informative reason in the dlqDeliveryFailureCause
attribute :)
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ DLQ issues
Posted by Tim Bain <tb...@alumni.duke.edu>.
First, the categorical statement is that when you hit an Out of memory
error, you have no guarantee that your application is still in a valid
state, so you should terminate the JVM and restart. So my first question
is, does this happen when you haven't run the JVM out of heap space? If
not, the behavior you're seeing isn't a bug. And if this occurred in an
operational broker and you haven't restarted it yet, that should be your
first order of business.
With that said, I'll try to address your other questions, in case the
behavior you're seeing turns out to not be related to the OOM you saw.
I agree with your statement that in your case, messages were not discarded
because they expired, since a 0 value for the expiration means that
feature's not in use for this mesages.
One way for non-expired messages to be discarded is to use the strategies
listed on http://activemq.apache.org/slow-consumer-handling.html. Are you
applying any of those configurations? (Since neither you nor Mike posted
your full config files, it's hard to know what you're using and what you're
not.)
Those strategies don't result in the creation an actual queue data
structure that run out of space, but they do allow the application of
limits on the number of messages that can be pending for a given consumer.
So the specific situation you suggested isn't possible, but there are code
paths that will discard messages when there are too many. However, those
are only enabled if you explicitly configure them, hence the question about
how you've configured the broker.
On a different note, from what you said, you're statically including a
virtual topic, which will cause its messages to be sent to the remote
broker irrespective of whether there are any consumers there. So, for the
topic in question, were there any consumers for that topic on the remote
broker? Could this have been as simple as a message being discarded because
there were no consumers once the message arrived on the remote broker? For
normal topics, messages discarded for that reason wouldn't go to the DLQ,
but I've not used virtual topics so I don't know whether that's a
possibility with them, so this question might be irrelevant.
Tim
On Nov 17, 2017 4:11 AM, "augustl" <au...@augustl.com> wrote:
> I also got these messages when I had a network connector setup. The setup
> seems similar to OP, where we had a message configured to be sent via a
> staticallyIncludedDestinations block, and it ended up in the DLQ with
> "TopicSubDiscard". Here's a full dump of one of the messages that ended up
> in the DLQ:
>
> <http://activemq.2283324.n4.nabble.com/file/t378787/
> Screenshot_from_2017-11-17_12-05-12.png>
>
> As you can see, message expiration was set to 0. So it seems like there was
> something else that casued it to expire.
>
> I noticed in the source code that there's another way for this error to
> occur:
>
> https://github.com/apache/activemq/blob/08aa5118ffcd8dfc4ec813d799a61e
> f1ee4421d1/activemq-broker/src/main/java/org/apache/
> activemq/broker/region/TopicSubscription.java#L189
>
> Seems like some sort of discarding of old messages when consumption is
> slow?
>
> Why does this happen to messages sent to a topic? Is there some sort of
> internal queue that's created for delivering messages over a network
> connector, and maybe this queue filled up somehow?
>
> By the way, we also got a lot of these in the logs of the activemq with the
> network connector where the message originated from and where it ended up
> in
> the DLQ:
>
> 2017-11-17 07:04:21,750 | ERROR | Checkpoint failed |
> org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
> Checkpoint Worker
> java.lang.OutOfMemoryError: Java heap space
> 2017-11-17 07:04:21,750 | INFO | Ignoring no space left exception,
> java.io.IOException: Java heap space |
> org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Journal
> Checkpoint Worker
> java.io.IOException: Java heap space
> at
> org.apache.activemq.util.IOExceptionSupport.create(
> IOExceptionSupport.java:40)[activemq-client-5.14.4.jar:5.14.4]
> at
> org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run(
> MessageDatabase.java:433)[activemq-kahadb-store-5.14.4.jar:5.14.4]
> at
> java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)[:1.8.0_144]
> at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_
> 144]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.
> java:180)[:1.8.0_144]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:294)[:1.8.0_144]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)[:1.8.0_144]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)[:1.8.0_144]
> at java.lang.Thread.run(Thread.java:748)[:1.8.0_144]
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Re: ActiveMQ DLQ issues
Posted by augustl <au...@augustl.com>.
I also got these messages when I had a network connector setup. The setup
seems similar to OP, where we had a message configured to be sent via a
staticallyIncludedDestinations block, and it ended up in the DLQ with
"TopicSubDiscard". Here's a full dump of one of the messages that ended up
in the DLQ:
<http://activemq.2283324.n4.nabble.com/file/t378787/Screenshot_from_2017-11-17_12-05-12.png>
As you can see, message expiration was set to 0. So it seems like there was
something else that casued it to expire.
I noticed in the source code that there's another way for this error to
occur:
https://github.com/apache/activemq/blob/08aa5118ffcd8dfc4ec813d799a61ef1ee4421d1/activemq-broker/src/main/java/org/apache/activemq/broker/region/TopicSubscription.java#L189
Seems like some sort of discarding of old messages when consumption is slow?
Why does this happen to messages sent to a topic? Is there some sort of
internal queue that's created for delivering messages over a network
connector, and maybe this queue filled up somehow?
By the way, we also got a lot of these in the logs of the activemq with the
network connector where the message originated from and where it ended up in
the DLQ:
2017-11-17 07:04:21,750 | ERROR | Checkpoint failed |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal
Checkpoint Worker
java.lang.OutOfMemoryError: Java heap space
2017-11-17 07:04:21,750 | INFO | Ignoring no space left exception,
java.io.IOException: Java heap space |
org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Journal
Checkpoint Worker
java.io.IOException: Java heap space
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:40)[activemq-client-5.14.4.jar:5.14.4]
at
org.apache.activemq.store.kahadb.MessageDatabase$CheckpointRunner.run(MessageDatabase.java:433)[activemq-kahadb-store-5.14.4.jar:5.14.4]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_144]
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)[:1.8.0_144]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)[:1.8.0_144]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)[:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)[:1.8.0_144]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)[:1.8.0_144]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_144]
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html