Resequencing on many individual topics (1000's)

[Joshua Watkins <jo...@gamesys.co.uk>]

I have run into a bit of a pickle. I am trying to use camel and Activemq to resequence messages off of thousands of individual topics. A particular topic would have a sequence number in messages delivered to it for that topic. Additionally, I don't know the specific topics to listen to ahead of time. So, at run time I will dynamically need to listen to NEWS.US.IDAHO.BOISE, or NEWS.EU.SPAIN.MADRID.

Unfortunately, since the resequencer can't work with multiple streams of messages I can't simply subscribe to NEWS.> or a subtopic. (This is a bit undesirable anyway as I would potentially be getting messages that I am not interested in.)

I have also tried simply creating a route dynamically with the specific topic and its resequencer. This however led to thousands of threads being created (1 thread for the JMS consumer and 2 threads for each resequencer).

Then I thought it would be better to use mildly more specific routes (NEWS.EU.SPAIN.> ) and use a filter to funnel a particular stream of messages to the resequencer. This ran into the problem of missing out on messages to the topic while I was altering the route. Retroactive consumer support doesn't seem to be supported in the Activemq component. (And I am guessing that I was right to believe that routes must be removed and re-added to the CamelContext to take effect.)

So, am I missing an easy solution for this?
Is there a way for me to pool JMS topic consumer threads somehow?
Can live camel routes be altered at runtime without them missing topic messages?

Thanks,
Josh

Re: Resequencing on many individual topics (1000's)

[Joshua Watkins <jo...@gamesys.co.uk>]

I have figure out a way to limit the number of threads per JMS consumer. I
needed to  use the JMSConfiguration pojo to configure my
ActiveMQComponent. The not-so-apparent change was changing the
consumerType to simple instead of default.

Hopefully this helps anyone else with the same issue.

-josh


On 31/03/2011 11:48, "Joshua Watkins" <jo...@gamesys.co.uk> wrote:

>I have run into a bit of a pickle. I am trying to use camel and Activemq
>to resequence messages off of thousands of individual topics. A
>particular topic would have a sequence number in messages delivered to it
>for that topic. Additionally, I don't know the specific topics to listen
>to ahead of time. So, at run time I will dynamically need to listen to
>NEWS.US.IDAHO.BOISE, or NEWS.EU.SPAIN.MADRID.
>
>Unfortunately, since the resequencer can't work with multiple streams of
>messages I can't simply subscribe to NEWS.> or a subtopic. (This is a bit
>undesirable anyway as I would potentially be getting messages that I am
>not interested in.)
>
>I have also tried simply creating a route dynamically with the specific
>topic and its resequencer. This however led to thousands of threads being
>created (1 thread for the JMS consumer and 2 threads for each
>resequencer).
>
>Then I thought it would be better to use mildly more specific routes
>(NEWS.EU.SPAIN.> ) and use a filter to funnel a particular stream of
>messages to the resequencer. This ran into the problem of missing out on
>messages to the topic while I was altering the route. Retroactive
>consumer support doesn't seem to be supported in the Activemq component.
>(And I am guessing that I was right to believe that routes must be
>removed and re-added to the CamelContext to take effect.)
>
>So, am I missing an easy solution for this?
>Is there a way for me to pool JMS topic consumer threads somehow?
>Can live camel routes be altered at runtime without them missing topic
>messages?
>
>Thanks,
>Josh