You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by swone <st...@hotmail.com> on 2015/10/29 22:04:52 UTC
ActiveMQ Topic Messages Not Delivered
Using an out-the-box installation of
jboss-fuse-pre-patched-6.2.0.redhat-143-p1 message sent to Topics are not
always delivered. Everything worked fine for a couple of weeks. Then this
error started happening. In one particular instance, the management console
showed 4 topics related to my topic - 3 advisory topics, Consumer, Expired
and Producer and the topic I created. Sending a message to to topic caused
the "Producer" Enqueue # to increase by 2. The Consumer Enqueue # remained
unchanged. The topic I created showed 2 Consumers with no message Enqueued
or Dequeued. Any ideas how to debug this?
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Topic-Messages-Not-Delivered-tp4703502.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Topic Messages Not Delivered
Posted by Tim Bain <tb...@alumni.duke.edu>.
In the default configuration, ActiveMQ will auto-create a destination when
a consumer subscribes to it or when a message is produced to it, whichever
happens first. Brokers in networks of brokers subscribe to the consumer
advisory topics (so they know where to route messages) but don't subscribe
to the other two, so those will be created only when messages are published
to them.
On Nov 2, 2015 10:23 AM, "swone" <st...@hotmail.com> wrote:
> Tim Bain wrote
> > Re: #1: Do you remember what the other two advisory topic names were?
>
> The other two I was seeing were ActiveMQ.Advisory.Producer.Topic.xxxxx and
> ActiveMQ.Adivsory.Expire.Topic.xxxxx . Looking at things this morning I'm
> also seeing the "Producer" topic as well. Maybe these topics only appear
> when the are being used?
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Topic-Messages-Not-Delivered-tp4703502p4703599.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ Topic Messages Not Delivered
Posted by swone <st...@hotmail.com>.
Tim Bain wrote
> Re: #1: Do you remember what the other two advisory topic names were?
The other two I was seeing were ActiveMQ.Advisory.Producer.Topic.xxxxx and
ActiveMQ.Adivsory.Expire.Topic.xxxxx . Looking at things this morning I'm
also seeing the "Producer" topic as well. Maybe these topics only appear
when the are being used?
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Topic-Messages-Not-Delivered-tp4703502p4703599.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Topic Messages Not Delivered
Posted by Tim Bain <tb...@alumni.duke.edu>.
I'm glad you were able to resolve the issue; it's possible you simply ran
out of space in your KahaDB instance but didn't have the limits set to keep
you from running off the edge, though I'm surprised that didn't result in
exceptions in the logs if that's really what it was. Seems strange.
Re: #1: Do you remember what the other two advisory topic names were? I've
seen topics for producers, consumers, and expired messages, and each is
created when someone subscribes to it or someone publishes to it (whichever
happens first), so it's possible that the other two haven't happened yet.
(There are two ways of publishing messages, one that creates a publisher
advisory and one that doesn't, and I think that JMSTemplate uses the one
that doesn't, so that makes sense for what you're seeing.)
Re: #2: I have no idea, but maybe someone else does.
Re: #3: I've never done anything with Fuse, so I have no idea, but maybe
someone else does.
Re: #4: Unless Fuse does something strange, an ActiveMQ broker generally
writes to only one log, so I don't think there would be any other log files
you'd need to look in.
Tim
On Fri, Oct 30, 2015 at 9:51 AM, swone <st...@hotmail.com> wrote:
> See below for answer to the questions. Here's some more information on
> changes made that seem to have gotten things working. The logs showed a
> message -
>
> Getting Worried {org.apache.activemq.FreeDiskSpaceLeft: WARNING Store limit
> is 33205 mb, whilst the data directory:
>
> /opt/jboss-fuse-pre-patched-6.2.0.redhat-143-p1/instances/dg-amq-1/data/dg-amq-1/kahadb
> only has 28337 mb of usable space from
>
> KahaDBPersistenceAdapter[/opt/jboss-fuse-pre-patched-6.2.0.redhat-143-p1/instances/dg-amq-1/data/dg-amq-1/kahadb]
> , }
>
> Other threads indicated this was a problem with the systemUsage parameters
> in the activemq.xml. Which I changed -
>
> *Original*
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> *Modification*
> <storeUsage>
> <storeUsage limit="20 mb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="20 mb"/>
> </tempUsage>
>
> After making these changes, I bounced the server and restarted my processes
> and message started flowing again.
>
> <
> http://activemq.2283324.n4.nabble.com/file/n4703531/activemqtopic_working.png
> >
>
> A few things I find puzzling, which makes me think I'm just missing
> something :
>
> 1) When messages were not being delivered, there were 4 Topics, 3 created
> by
> ActiveMQ and 1 created by the Producer. When messages are being delivered,
> there are only 2. Why is that?
>
> 2) The warning message indicated the store limit as 33205 mb, whereas
> activemq.xml had it specified as 100 gb. I also wasn't able to located any
> configuration files containing a "mb" specification. Why the disconnect?
>
> 3) The activemq.xml file I changed was located at $FUSE_HOME/etc. The
> documentation indicates this is used as a template when creating new
> instances. In $FUSE_HOME/instances/amq-1/etc the file doesn't exist. I
> would
> have expected the template file would have been added to the instance
> directory and then the instance could be tweaked independently of other
> instances. Does this mean all activemq instances will use the same
> activemq.xml?
>
> 4) When messages are not delivered, I found nothing in the logs indicating
> the failure. There was just the one warning message. I looked in
> $FUSE_HOME/data/log and $FUSE_HOME/instances/amq-1/data/log. Are there
> other
> log directories that I may be missing?
>
> ------------------------
> *Replies to e-mail questions*
> Stepping through the code in the debugger show a successful publish of the
> message. Also running in Cloud Foundry produced the log entry that the
> message was successfully published.
>
>
> Tim Bain wrote
> > Also, do I understand correctly that "2 Consumers with no message
> Enqueued
> > or Dequeued" means no additional messages enqueued/dequeued once the
> > problems start, not that the number is actually 0?
>
> This is correct. No message enqueued/dequeue once the issue started.
>
>
> Tim Bain wrote
> > Am I right to assume that if there were any errors in either the broker
> > logs or the producer logs, you'd have mentioned them?
>
> Correct. The only log message was the aforementioned warning.
>
> *Java / Spring*
>
> // Same code snippet in Producer and Consumer to create the factory
> ActiveMQConnectionFactory activeMQConnectionFactory = new
> ActiveMQConnectionFactory();
>
>
>
> activeMQConnectionFactory.setBrokerURL(environment.getProperty("jms.host"));
>
>
> activeMQConnectionFactory.setUserName(environment.getProperty("jms.username"));
>
>
> activeMQConnectionFactory.setPassword(environment.getProperty("jms.password"));
>
> *Publisher *
>
> // Publishing snippet
> logger.debug("Topic : " + topicName);
> logger.debug("Queuing message: " + message);
>
> ObjectMapper mapper = new ObjectMapper();
> mapper.setSerializationInclusion(Include.NON_NULL);
>
> try
> {
> jmsTemplate.setPubSubDomain(true);
> jmsTemplate.setExplicitQosEnabled(true);
> jmsTemplate.setTimeToLive(MESSAGE_TTL);
> jmsTemplate.convertAndSend(topicName,
> mapper.writeValueAsString(message));
>
> logger.debug("Successfully sent to : " + topicName);
>
> *Consumer*
>
> // Snippet creating listener/consumer
> DefaultMessageListenerContainer messageListener = new
> DefaultMessageListenerContainer();
>
> messageListener.setSessionTransacted(true);
> messageListener.setDestinationName(destinationName);
> messageListener.setConnectionFactory(jmsFactory);
> messageListener.setMessageListener(listener);
> messageListener.setPubSubDomain(true);
> messageListener.initialize();
> messageListener.start();
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Topic-Messages-Not-Delivered-tp4703502p4703531.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ Topic Messages Not Delivered
Posted by swone <st...@hotmail.com>.
See below for answer to the questions. Here's some more information on
changes made that seem to have gotten things working. The logs showed a
message -
Getting Worried {org.apache.activemq.FreeDiskSpaceLeft: WARNING Store limit
is 33205 mb, whilst the data directory:
/opt/jboss-fuse-pre-patched-6.2.0.redhat-143-p1/instances/dg-amq-1/data/dg-amq-1/kahadb
only has 28337 mb of usable space from
KahaDBPersistenceAdapter[/opt/jboss-fuse-pre-patched-6.2.0.redhat-143-p1/instances/dg-amq-1/data/dg-amq-1/kahadb]
, }
Other threads indicated this was a problem with the systemUsage parameters
in the activemq.xml. Which I changed -
*Original*
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
*Modification*
<storeUsage>
<storeUsage limit="20 mb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="20 mb"/>
</tempUsage>
After making these changes, I bounced the server and restarted my processes
and message started flowing again.
<http://activemq.2283324.n4.nabble.com/file/n4703531/activemqtopic_working.png>
A few things I find puzzling, which makes me think I'm just missing
something :
1) When messages were not being delivered, there were 4 Topics, 3 created by
ActiveMQ and 1 created by the Producer. When messages are being delivered,
there are only 2. Why is that?
2) The warning message indicated the store limit as 33205 mb, whereas
activemq.xml had it specified as 100 gb. I also wasn't able to located any
configuration files containing a "mb" specification. Why the disconnect?
3) The activemq.xml file I changed was located at $FUSE_HOME/etc. The
documentation indicates this is used as a template when creating new
instances. In $FUSE_HOME/instances/amq-1/etc the file doesn't exist. I would
have expected the template file would have been added to the instance
directory and then the instance could be tweaked independently of other
instances. Does this mean all activemq instances will use the same
activemq.xml?
4) When messages are not delivered, I found nothing in the logs indicating
the failure. There was just the one warning message. I looked in
$FUSE_HOME/data/log and $FUSE_HOME/instances/amq-1/data/log. Are there other
log directories that I may be missing?
------------------------
*Replies to e-mail questions*
Stepping through the code in the debugger show a successful publish of the
message. Also running in Cloud Foundry produced the log entry that the
message was successfully published.
Tim Bain wrote
> Also, do I understand correctly that "2 Consumers with no message Enqueued
> or Dequeued" means no additional messages enqueued/dequeued once the
> problems start, not that the number is actually 0?
This is correct. No message enqueued/dequeue once the issue started.
Tim Bain wrote
> Am I right to assume that if there were any errors in either the broker
> logs or the producer logs, you'd have mentioned them?
Correct. The only log message was the aforementioned warning.
*Java / Spring*
// Same code snippet in Producer and Consumer to create the factory
ActiveMQConnectionFactory activeMQConnectionFactory = new
ActiveMQConnectionFactory();
activeMQConnectionFactory.setBrokerURL(environment.getProperty("jms.host"));
activeMQConnectionFactory.setUserName(environment.getProperty("jms.username"));
activeMQConnectionFactory.setPassword(environment.getProperty("jms.password"));
*Publisher *
// Publishing snippet
logger.debug("Topic : " + topicName);
logger.debug("Queuing message: " + message);
ObjectMapper mapper = new ObjectMapper();
mapper.setSerializationInclusion(Include.NON_NULL);
try
{
jmsTemplate.setPubSubDomain(true);
jmsTemplate.setExplicitQosEnabled(true);
jmsTemplate.setTimeToLive(MESSAGE_TTL);
jmsTemplate.convertAndSend(topicName,
mapper.writeValueAsString(message));
logger.debug("Successfully sent to : " + topicName);
*Consumer*
// Snippet creating listener/consumer
DefaultMessageListenerContainer messageListener = new
DefaultMessageListenerContainer();
messageListener.setSessionTransacted(true);
messageListener.setDestinationName(destinationName);
messageListener.setConnectionFactory(jmsFactory);
messageListener.setMessageListener(listener);
messageListener.setPubSubDomain(true);
messageListener.initialize();
messageListener.start();
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Topic-Messages-Not-Delivered-tp4703502p4703531.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Topic Messages Not Delivered
Posted by Tim Bain <tb...@alumni.duke.edu>.
Can you post your publisher code? Have you stepped through it with a
debugger to confirm that the publish is happening without errors or
exceptions (if it uses a technology that's conducive to debuggers)?
Also, do I understand correctly that "2 Consumers with no message Enqueued
or Dequeued" means no additional messages enqueued/dequeued once the
problems start, not that the number is actually 0?
Am I right to assume that if there were any errors in either the broker
logs or the producer logs, you'd have mentioned them?
Tim
On Oct 29, 2015 3:17 PM, "swone" <st...@hotmail.com> wrote:
> Using an out-the-box installation of
> jboss-fuse-pre-patched-6.2.0.redhat-143-p1 message sent to Topics are not
> always delivered. Everything worked fine for a couple of weeks. Then this
> error started happening. In one particular instance, the management console
> showed 4 topics related to my topic - 3 advisory topics, Consumer, Expired
> and Producer and the topic I created. Sending a message to to topic caused
> the "Producer" Enqueue # to increase by 2. The Consumer Enqueue # remained
> unchanged. The topic I created showed 2 Consumers with no message Enqueued
> or Dequeued. Any ideas how to debug this?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Topic-Messages-Not-Delivered-tp4703502.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>