You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@camel.apache.org by ddewaele <dd...@gmail.com> on 2014/10/21 12:16:05 UTC

Camel JMS message topic handling

We were testing ActiveMQ topics with Camel and noticed the following
behaviour:

- Our Camel route was up and running with an activemq topic consumer.
- Messages were published on the topic whle the route is active.
- our Camel JMS consumer that was listening on that topic didn't receive all
messages.

We were not using Durable JMS topics or ActiveMQ Virtual Topics but I did
find it strange that it would drop messages here.

I assume these JMS Consumers are connected to the JMS broker while messages
are put on the topic and that all messages would be properly processed. Yet
we noticed a lot of drops.

When using ActiveMQ virtual topics we don't experience this issue and the
Camel route is always able to process all messages.

I was just wondering why this doesn't work with a plain vanilla topic and an
active Camel route.



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by ddewaele <dd...@gmail.com>.

That's true, but in Camel a Spring JMS
AbstractPollingMessageListenerContainer implementation is used. After the
thread finished processing the message, it does this:

		finally {
			JmsUtils.closeMessageConsumer(consumerToClose);
			JmsUtils.closeSession(sessionToClose);
			ConnectionFactoryUtils.releaseConnection(conToClose,
getConnectionFactory(), true);
		}

According to me the consumer / session is closed and this consumer will no
longer do anything anymore (goes offline like you say)
.
If by the time another consumer was brought up by the
AbstractPollingMessageListenerContainer (via the createListenerConsumer
method) messages arrived on the topic they will be lost, as this new
consumer will be too late.

There is a durableSubscriptionName option on the camel activeMQ component
that might keep the consumer alive. That I haven't tried yet.

But the question remains, what is the use-case of having an out-of-the-box
jms consumer (non-durable subscription) on a standard (non virtual) activeMQ
topic





--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853p5758028.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by Minh Tran <da...@gmail.com>.

Durability is only applicable when the consumers go offline. In your case, the consumer never goes offline, JMS will guarantee delivery of the message to the subscriber regardless of how slow it is consuming any prior messages. If that consumer goes offline then that’s a different story.



On 25 Oct 2014, at 6:36 pm, ddewaele <dd...@gmail.com> wrote:

> I think messages will be delivered “at most once” and non-durable subscribers
> can miss messages if they are inactive. In the case of camel, it is is a
> polling consumer, meaning that
> 
> 1. it is started
> 2. it starts receiving messages (respecting timeouts) 
> 3. it is stopped (becomes inactive)
> 
> According to me, when the consumer is in step 2 (processing an incoming msg)
> this particular consumer will not receive any messages anymore. It needs to
> exit as soon as possible so another polling consumer can pick up the next
> one. If a message arrives in the meantime it will be lost.
> 
> That's my understanding of it at least in a plain vanilla 1 thread/consumer
> and plain topic setup ... please correct me if I'm wrong.
> 
> 
> 
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853p5758021.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by ddewaele <dd...@gmail.com>.

I think messages will be delivered “at most once” and non-durable subscribers
can miss messages if they are inactive. In the case of camel, it is is a
polling consumer, meaning that

1. it is started
2. it starts receiving messages (respecting timeouts) 
3. it is stopped (becomes inactive)

According to me, when the consumer is in step 2 (processing an incoming msg)
this particular consumer will not receive any messages anymore. It needs to
exit as soon as possible so another polling consumer can pick up the next
one. If a message arrives in the meantime it will be lost.

That's my understanding of it at least in a plain vanilla 1 thread/consumer
and plain topic setup ... please correct me if I'm wrong.



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853p5758021.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by Minh Tran <da...@gmail.com>.

I’m not sure how it all works internally but consumers on topics do not lose incoming messages while it is busy processing one message. That would break the rule of JMS guaranteed message delivery. 

I’ve had a number of projects using a topic with only one consumer on it. Normally the topic is chosen because we need to broadcast the message out and currently there’s only one interested subscriber but potentially we could add more in the future. Perfectly valid use case.

On 25 Oct 2014, at 7:32 am, ddewaele <dd...@gmail.com> wrote:

> As there is only 1 thread where a JMS Consumer is active (to avoid duplicate
> msg processing), isn't it normal that messages get lost in this scenario ?
> 
> When that thread is busy processing a message
> (DefaultJmsMessageListenerContainer(AbstractPollingMessageListenerContainer).doReceiveAndExecute),
> any message that comes while this JMSConsumer is active will be lost, as
> this thread will close the message consumer when it's done.
> 
> Having multiple active consumers would not help and would even make matters
> worse , as you would need to deal with duplicate messages.
> 
> In what use-case would a basic (non-virtual) activeMQ topic combined with a
> single JMS consumer make sense ? 
> 
> 
> 
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853p5758004.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by ddewaele <dd...@gmail.com>.

As there is only 1 thread where a JMS Consumer is active (to avoid duplicate
msg processing), isn't it normal that messages get lost in this scenario ?

When that thread is busy processing a message
(DefaultJmsMessageListenerContainer(AbstractPollingMessageListenerContainer).doReceiveAndExecute),
any message that comes while this JMSConsumer is active will be lost, as
this thread will close the message consumer when it's done.

Having multiple active consumers would not help and would even make matters
worse , as you would need to deal with duplicate messages.

In what use-case would a basic (non-virtual) activeMQ topic combined with a
single JMS consumer make sense ? 



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853p5758004.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by ddewaele <dd...@gmail.com>.

Interesting.... we had a similar issue with the MQTT endpoint

http://camel.465427.n5.nabble.com/Camel-Endpoint-Consumer-startup-td5757635.html

There the endpoint was also started (causing TCP connections to be
established) before any consumers became active (resulting in loss of
messages).

I'll try to debug this further today



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853p5757878.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by Minh Tran <da...@gmail.com>.

I found this similar behaviour as well. 

http://mail-archives.apache.org/mod_mbox/camel-users/201404.mbox/%3CF54F1C39-DD8A-4C17-A51D-C564F3230911@winning.com.au%3E

Appears as though the topic subscription is not guaranteed to happen before the camel context finishes starting up. So if the message is published before the subscription occurs, the message gets sent to nobody as per usual topic subscription rules.

The unit test in camel puts in a thread sleep to get around the problem but IMO adding a sleep to get around race conditions are a poor solution. 

// give a bit of time for AMQ to properly setup topic subscribers
Thread.sleep(500);

I’m unsure if it’s a camel or an AMQ bug but I think the topic subscription should be guaranteed before the camel context has started up.

On 21 Oct 2014, at 9:16 pm, ddewaele <dd...@gmail.com> wrote:

> We were testing ActiveMQ topics with Camel and noticed the following
> behaviour:
> 
> - Our Camel route was up and running with an activemq topic consumer.
> - Messages were published on the topic whle the route is active.
> - our Camel JMS consumer that was listening on that topic didn't receive all
> messages.
> 
> We were not using Durable JMS topics or ActiveMQ Virtual Topics but I did
> find it strange that it would drop messages here.
> 
> I assume these JMS Consumers are connected to the JMS broker while messages
> are put on the topic and that all messages would be properly processed. Yet
> we noticed a lot of drops.
> 
> When using ActiveMQ virtual topics we don't experience this issue and the
> Camel route is always able to process all messages.
> 
> I was just wondering why this doesn't work with a plain vanilla topic and an
> active Camel route.
> 
> 
> 
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by ddewaele <dd...@gmail.com>.

We're using the Camel ActiveMQ endpoint
(http://camel.apache.org/activemq.html) and have setup a consumer in our
route via the following URI : *activemq:topic:TopicName*.







--
View this message in context: http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853p5757857.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Camel JMS message topic handling

Posted by Charles Moulliard <ch...@gmail.com>.

Hi,

We have different unit tests within the Apache Camel project using a Topic
and they work perfectly
Which component do you use (camel-activemq, camel-jms or camel-sjms) ?

Examples :
-
https://github.com/apache/camel/blob/master/components/camel-jms/src/test/java/org/apache/camel/component/jms/JmsRouteWithInOnlyTest.java
-
https://github.com/apache/camel/blob/master/components/camel-jms/src/test/java/org/apache/camel/component/jms/TwoConsumerOnSameTopicTest.java

Regards,


On Tue, Oct 21, 2014 at 12:16 PM, ddewaele <dd...@gmail.com> wrote:

> We were testing ActiveMQ topics with Camel and noticed the following
> behaviour:
>
> - Our Camel route was up and running with an activemq topic consumer.
> - Messages were published on the topic whle the route is active.
> - our Camel JMS consumer that was listening on that topic didn't receive
> all
> messages.
>
> We were not using Durable JMS topics or ActiveMQ Virtual Topics but I did
> find it strange that it would drop messages here.
>
> I assume these JMS Consumers are connected to the JMS broker while messages
> are put on the topic and that all messages would be properly processed. Yet
> we noticed a lot of drops.
>
> When using ActiveMQ virtual topics we don't experience this issue and the
> Camel route is always able to process all messages.
>
> I was just wondering why this doesn't work with a plain vanilla topic and
> an
> active Camel route.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Camel-JMS-message-topic-handling-tp5757853.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>



-- 
Charles Moulliard
Apache Committer / Architect @RedHat
Twitter : @cmoulliard | Blog :  http://cmoulliard.github.io