Consuming messages from Kafka and pushing on to a JMS queue

[Andrew Longwill <an...@openbet.com>]

Hi,

We have an application that currently uses Camel to forward a JMS message
from HornetQ to multiple consumers (JMS queues).

The one challenge we have is handling failure of one of the consumers.
Kafka seems to provide a solution here by allowing different consumers to
keep track of their own offset.

However we'd prefer to ultimately "push" the messages to each end point and
not use Kafka's "pull" model as it requires implementing the Kafka's
Consumer API. One approach could be to write  intermediary consumers which
forward each message onto the appropriate JMS queue. Is this a good
approach or is there a better way to do this (e.g. use Apache Storm)?

Thanks

Andrew

Re: Consuming messages from Kafka and pushing on to a JMS queue

[Neha Narkhede <ne...@gmail.com>]

The power of the consumption APIs and the general performance offered by
Kafka is only useful if you can send the data to Kafka and use the consumer
APIs. Apache Storm will not solve the problem if you are trying to avoid
using the Kafka consumer APIs. I would rethink the architecture you
currently have and see if the JMS queues can be replaced by Kafka
consumers. This would require a plugin that could pull data out of the
source JMS queue, if there isn't a way to replace that as well.


On Wed, Aug 13, 2014 at 6:06 AM, Andrew Longwill <
andrew.longwill@openbet.com> wrote:

> Hi,
>
> We have an application that currently uses Camel to forward a JMS message
> from HornetQ to multiple consumers (JMS queues).
>
> The one challenge we have is handling failure of one of the consumers.
> Kafka seems to provide a solution here by allowing different consumers to
> keep track of their own offset.
>
> However we'd prefer to ultimately "push" the messages to each end point and
> not use Kafka's "pull" model as it requires implementing the Kafka's
> Consumer API. One approach could be to write  intermediary consumers which
> forward each message onto the appropriate JMS queue. Is this a good
> approach or is there a better way to do this (e.g. use Apache Storm)?
>
> Thanks
>
> Andrew
>