You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Vishwakarma, Chhaya" <Ch...@Teradata.com> on 2015/05/07 10:50:58 UTC
Getting data from IBM MQ to Hadoop
Hi All,
I want to read data from IBM MQ and put it into HDFs.
Looked into JMS source of flume, seems it can connect to IBM MQ, but I'm not understanding what does "destinationType" and "destinationName" mean in the list of required properties. Can someone please explain?
Also, how I should be configuring my flume agents
flumeAgent1(runs on the machine same as MQ) reads MQ data ------> flumeAgent2(Runs on Hadoop cluster) writes into Hdfs
OR only one agent is enough on Hadoop cluster
Can someone help me in understanding how MQs can be integrated with flume
Thanks,
Chhaya
RE: Getting data from IBM MQ to Hadoop
Posted by Gwen Shapira <gs...@cloudera.com>.
Yep.
On May 8, 2015 4:49 PM, "Vishwakarma, Chhaya" <
Chhaya.Vishwakarma@teradata.com> wrote:
> Hi Gwen,
>
>
>
> Thanks for the reply
>
>
>
> a1.sources.r1.destinationName *=* BUSINESS_DATA
>
> In above configuration “BUSINESS_DATA” is name of a queue?
>
>
>
> *From:* Gwen Shapira [mailto:gshapira@cloudera.com]
> *Sent:* Thursday, May 07, 2015 2:57 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Getting data from IBM MQ to Hadoop
>
>
>
> Hi Chhaya,
>
>
>
> First, it looks like one agent should be enough. Don't run agents on the
> Hadoop cluster itself (i.e not on data nodes). You can give it its own
> machine, share it with other "edge node" services (like Hue) or install it
> on MQ machine (if the machine is not too busy).
>
>
>
> Second, destination should have probably been named "source", i.e. thats
> the queue or topic that contains the data in JMS.
>
>
>
> There is a nice example in the docs:
>
> a1.sources *=* r1
>
> a1.channels *=* c1
>
> a1.sources.r1.type *=* jms
>
> a1.sources.r1.channels *=* c1
>
> a1.sources.r1.initialContextFactory *=* org.apache.activemq.jndi.ActiveMQInitialContextFactory
>
> a1.sources.r1.connectionFactory *=* GenericConnectionFactory
>
> a1.sources.r1.providerURL *=* tcp://mqserver:61616
>
> a1.sources.r1.destinationName *=* BUSINESS_DATA
>
> a1.sources.r1.destinationType *=* QUEUE
>
>
>
> On Thu, May 7, 2015 at 1:50 AM, Vishwakarma, Chhaya <
> Chhaya.Vishwakarma@teradata.com> wrote:
>
> Hi All,
>
>
>
> I want to read data from IBM MQ and put it into HDFs.
>
>
>
> Looked into JMS source of flume, seems it can connect to IBM MQ, but I’m
> not understanding what does “destinationType” and “destinationName” mean in
> the list of required properties. Can someone please explain?
>
>
>
> Also, how I should be configuring my flume agents
>
>
>
> flumeAgent1(runs on the machine same as MQ) reads MQ data ----à
> flumeAgent2(Runs on Hadoop cluster) writes into Hdfs
>
> OR only one agent is enough on Hadoop cluster
>
>
>
> Can someone help me in understanding how MQs can be integrated with flume
>
>
>
> Thanks,
>
> Chhaya
>
>
>
>
>
RE: Getting data from IBM MQ to Hadoop
Posted by "Vishwakarma, Chhaya" <Ch...@Teradata.com>.
Hi Gwen,
Thanks for the reply
a1.sources.r1.destinationName = BUSINESS_DATA
In above configuration “BUSINESS_DATA” is name of a queue?
From: Gwen Shapira [mailto:gshapira@cloudera.com]
Sent: Thursday, May 07, 2015 2:57 PM
To: user@flume.apache.org
Subject: Re: Getting data from IBM MQ to Hadoop
Hi Chhaya,
First, it looks like one agent should be enough. Don't run agents on the Hadoop cluster itself (i.e not on data nodes). You can give it its own machine, share it with other "edge node" services (like Hue) or install it on MQ machine (if the machine is not too busy).
Second, destination should have probably been named "source", i.e. thats the queue or topic that contains the data in JMS.
There is a nice example in the docs:
a1.sources = r1
a1.channels = c1
a1.sources.r1.type = jms
a1.sources.r1.channels = c1
a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
a1.sources.r1.connectionFactory = GenericConnectionFactory
a1.sources.r1.providerURL = tcp://mqserver:61616
a1.sources.r1.destinationName = BUSINESS_DATA
a1.sources.r1.destinationType = QUEUE
On Thu, May 7, 2015 at 1:50 AM, Vishwakarma, Chhaya <Ch...@teradata.com>> wrote:
Hi All,
I want to read data from IBM MQ and put it into HDFs.
Looked into JMS source of flume, seems it can connect to IBM MQ, but I’m not understanding what does “destinationType” and “destinationName” mean in the list of required properties. Can someone please explain?
Also, how I should be configuring my flume agents
flumeAgent1(runs on the machine same as MQ) reads MQ data ------> flumeAgent2(Runs on Hadoop cluster) writes into Hdfs
OR only one agent is enough on Hadoop cluster
Can someone help me in understanding how MQs can be integrated with flume
Thanks,
Chhaya
Re: Getting data from IBM MQ to Hadoop
Posted by Gwen Shapira <gs...@cloudera.com>.
Hi Chhaya,
First, it looks like one agent should be enough. Don't run agents on the
Hadoop cluster itself (i.e not on data nodes). You can give it its own
machine, share it with other "edge node" services (like Hue) or install it
on MQ machine (if the machine is not too busy).
Second, destination should have probably been named "source", i.e. thats
the queue or topic that contains the data in JMS.
There is a nice example in the docs:
a1.sources = r1a1.channels = c1a1.sources.r1.type =
jmsa1.sources.r1.channels = c1a1.sources.r1.initialContextFactory =
org.apache.activemq.jndi.ActiveMQInitialContextFactorya1.sources.r1.connectionFactory
= GenericConnectionFactorya1.sources.r1.providerURL =
tcp://mqserver:61616a1.sources.r1.destinationName =
BUSINESS_DATAa1.sources.r1.destinationType = QUEUE
On Thu, May 7, 2015 at 1:50 AM, Vishwakarma, Chhaya <
Chhaya.Vishwakarma@teradata.com> wrote:
> Hi All,
>
>
>
> I want to read data from IBM MQ and put it into HDFs.
>
>
>
> Looked into JMS source of flume, seems it can connect to IBM MQ, but I’m
> not understanding what does “destinationType” and “destinationName” mean in
> the list of required properties. Can someone please explain?
>
>
>
> Also, how I should be configuring my flume agents
>
>
>
> flumeAgent1(runs on the machine same as MQ) reads MQ data ----à
> flumeAgent2(Runs on Hadoop cluster) writes into Hdfs
>
> OR only one agent is enough on Hadoop cluster
>
>
>
> Can someone help me in understanding how MQs can be integrated with flume
>
>
>
> Thanks,
>
> Chhaya
>
>
>