You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Kajal K Das <ka...@in.ibm.com> on 2017/08/31 07:05:21 UTC
Looking for a specific input on JMS source
Hi All,
We are designing a near-real time pipeline in which we will receive
messages from IBM MQ. On the Hadoop side, we are planning to use Flume to
ingest the messages using JMS source.
The messages will include some custom header information, which we need to
extract as well. We discovered that interceptors cannot be configured with
a JMS source.
In that case: what are the alternatives in order to extract the message
headers using Flume? Or should we try something else (e.g. Kafka)?
Regards
Kajal Kr. Das
Re: Looking for a specific input on JMS source
Posted by Kajal K Das <ka...@in.ibm.com>.
Thank you, Muhammad. Much appreciated.
Regards
Kajal Kr. Das
Solution Architect - Etihad Ops BI
IBM
SUT #: +91 (0)80 4318 2743 | India Mobile #: +91 (0)98301 15013
E-mail: kajal.das@in.ibm.com
From: Muhammad Yaseen <ya...@gmail.com>
To: user@flume.apache.org
Date: 08/31/2017 12:47 PM
Subject: Re: Looking for a specific input on JMS source
Hello,
I think you can achieve what you want with the JMSSourceConverter. Looking
at the Flume documentation it says that "The JMS source allows pluggable
converters, though it?s likely the default converter will work for most
purposes. The default converter is able to convert Bytes, Text, and Object
messages to FlumeEvents. In all cases, the properties in the message are
added as headers to the FlumeEvent".
Perhaps you can implement the JSMMessageConverter interface (
https://github.com/apache/flume/blob/dfa0627573b9a75a25dc7149a7d63c9bac953ff4/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSMessageConverter.java
). The default converted also implements this interface (
https://github.com/apache/flume/blob/dfa0627573b9a75a25dc7149a7d63c9bac953ff4/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java
) and also provides a good reference point for implementing you own.
You custom code would go into plugins.d directory in flume. More details
here: https://flume.apache.org/FlumeUserGuide.html#jms-source
On Thu, Aug 31, 2017 at 12:05 PM, Kajal K Das <ka...@in.ibm.com>
wrote:
Hi All,
We are designing a near-real time pipeline in which we will receive
messages from IBM MQ. On the Hadoop side, we are planning to use Flume to
ingest the messages using JMS source.
The messages will include some custom header information, which we need to
extract as well. We discovered that interceptors cannot be configured with
a JMS source.
In that case: what are the alternatives in order to extract the message
headers using Flume? Or should we try something else (e.g. Kafka)?
Regards
Kajal Kr. Das
Re: Looking for a specific input on JMS source
Posted by Muhammad Yaseen <ya...@gmail.com>.
Hello,
I think you can achieve what you want with the JMSSourceConverter. Looking
at the Flume documentation it says that "The JMS source allows pluggable
converters, though it’s likely the default converter will work for most
purposes. The default converter is able to convert Bytes, Text, and Object
messages to FlumeEvents. In all cases, the properties in the message are
added as headers to the FlumeEvent".
Perhaps you can implement the JSMMessageConverter interface (
https://github.com/apache/flume/blob/dfa0627573b9a75a25dc7149a7d63c9bac953ff4/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSMessageConverter.java).
The default converted also implements this interface (
https://github.com/apache/flume/blob/dfa0627573b9a75a25dc7149a7d63c9bac953ff4/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java)
and also provides a good reference point for implementing you own.
You custom code would go into plugins.d directory in flume. More details
here: https://flume.apache.org/FlumeUserGuide.html#jms-source
On Thu, Aug 31, 2017 at 12:05 PM, Kajal K Das <ka...@in.ibm.com> wrote:
> Hi All,
>
> We are designing a near-real time pipeline in which we will receive
> messages from IBM MQ. On the Hadoop side, we are planning to use Flume to
> ingest the messages using JMS source.
> The messages will include some custom header information, which we need to
> extract as well. We discovered that interceptors cannot be configured with
> a JMS source.
> In that case: what are the alternatives in order to extract the message
> headers using Flume? Or should we try something else (e.g. Kafka)?
>
>
>
> Regards
> Kajal Kr. Das
>
>