You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Madabhattula Rajesh Kumar <mr...@gmail.com> on 2014/03/04 04:22:22 UTC
Flume JMS source
Hello Team,
I'm new to flume. Could you please help me on below query
1. How to configure JMS source(TIBCO and ActiveMQ) in flume?
2. Do i need to write custom source for above requirement? (Or) Out Of
the Box flume JMS source implementation is fine?
3. What is the best channel mechanism for above requirement?
4. Each message will be sink into HDFS? (or) Can we consolidate some
messages and sink into HDFS?
Thank you in advance
Regards,
Rajesh
Re: Flume JMS source
Posted by Richard Ross <ri...@gmail.com>.
Thats correct.
On Mar 6, 2014, at 1:09 PMEST, Madabhattula Rajesh Kumar wrote:
>
> Hello Richard,
>
> Thank you very much. I'll try this configuration. Could you please clarify below questions
>
> a1.sources.r1.providerURL = tcp://<activemqserver>:61616
> a1.sources.r1.destinationName = TESTQUEUE -----> This is queue name right?
> a1.sources.r1.destinationType = QUEUE
>
> Regards,
> Rajesh
>
>
> On Tue, Mar 4, 2014 at 8:13 PM, richard ross <ri...@gmail.com> wrote:
> Hello Rajesh:
>
> I just worked on a POC for the work that I am doing with JMS --> HDFS configuration using ActiveMQ. I was able to use everything right out of the box, going with JMS Source --> Memory Channel --> HDFS Sink, based on the guide (https://flume.apache.org/FlumeUserGuide.html). Below is my example config that I was playing around with. Note that for ActiveMQ, you do not want to specify the connectionFactory in the source configuration (https://flume.apache.org/FlumeUserGuide.html#jms-source). A documentation defect was created for this point (https://issues.apache.org/jira/browse/FLUME-2339). As for questions 3 and 4, I will let someone with more experience answer.
>
> # Name the components on this agent
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
>
> # Describe the source
> #a1.sources.r1.type = netcat
> #a1.sources.r1.bind = localhost
> #a1.sources.r1.port = 44444
> a1.sources.r1.type = jms
> a1.sources.r1.channels = c1
> a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
> a1.sources.r1.providerURL = tcp://<activemqserver>:61616
> a1.sources.r1.destinationName = TESTQUEUE
> a1.sources.r1.destinationType = QUEUE
>
> # Describe the sink
> #a1.sinks.k1.type = logger
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.path = hdfs://<namenode>:8020/user/root/flumetest/%Y%m%d
> a1.sinks.k1.hdfs.filePrefix = messages
> a1.sinks.k1.hdfs.rollInterval = 30
> #a1.sinks.k1.hdfs.rollSize = 1024
> a1.sinks.k1.hdfs.rollCount = 1000
> a1.sinks.k1.hdfs.fileType = DataStream
>
> # Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
>
> # Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
>
> Hope this helps,
> Richard.
>
>
> On Mon, Mar 3, 2014 at 10:22 PM, Madabhattula Rajesh Kumar <mr...@gmail.com> wrote:
> Hello Team,
>
> I'm new to flume. Could you please help me on below query
> How to configure JMS source(TIBCO and ActiveMQ) in flume?
> Do i need to write custom source for above requirement? (Or) Out Of the Box flume JMS source implementation is fine?
> What is the best channel mechanism for above requirement?
> Each message will be sink into HDFS? (or) Can we consolidate some messages and sink into HDFS?
> Thank you in advance
>
> Regards,
> Rajesh
>
>
Re: Flume JMS source
Posted by Madabhattula Rajesh Kumar <mr...@gmail.com>.
Hello Richard,
Thank you very much. I'll try this configuration. Could you please clarify
below questions
a1.sources.r1.providerURL = tcp://<activemqserver>:61616
a1.sources.r1.destinationName = TESTQUEUE -----> *This is queue name right?*
a1.sources.r1.destinationType = QUEUE
Regards,
Rajesh
On Tue, Mar 4, 2014 at 8:13 PM, richard ross <ri...@gmail.com>wrote:
> Hello Rajesh:
>
> I just worked on a POC for the work that I am doing with JMS --> HDFS
> configuration using ActiveMQ. I was able to use everything right out of the
> box, going with JMS Source --> Memory Channel --> HDFS Sink, based on the
> guide (https://flume.apache.org/FlumeUserGuide.html). Below is my example
> config that I was playing around with. Note that for ActiveMQ, you do not
> want to specify the connectionFactory in the source configuration (
> https://flume.apache.org/FlumeUserGuide.html#jms-source). A documentation
> defect was created for this point (
> https://issues.apache.org/jira/browse/FLUME-2339). As for questions 3 and
> 4, I will let someone with more experience answer.
>
> # Name the components on this agent
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
>
> # Describe the source
> #a1.sources.r1.type = netcat
> #a1.sources.r1.bind = localhost
> #a1.sources.r1.port = 44444
> a1.sources.r1.type = jms
> a1.sources.r1.channels = c1
> a1.sources.r1.initialContextFactory =
> org.apache.activemq.jndi.ActiveMQInitialContextFactory
> a1.sources.r1.providerURL = tcp://<activemqserver>:61616
> a1.sources.r1.destinationName = TESTQUEUE
> a1.sources.r1.destinationType = QUEUE
>
> # Describe the sink
> #a1.sinks.k1.type = logger
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.hdfs.path = hdfs://<namenode>:8020/user/root/flumetest/%Y%m%d
> a1.sinks.k1.hdfs.filePrefix = messages
> a1.sinks.k1.hdfs.rollInterval = 30
> #a1.sinks.k1.hdfs.rollSize = 1024
> a1.sinks.k1.hdfs.rollCount = 1000
> a1.sinks.k1.hdfs.fileType = DataStream
>
> # Use a channel which buffers events in memory
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
>
> # Bind the source and sink to the channel
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
>
> Hope this helps,
> Richard.
>
>
> On Mon, Mar 3, 2014 at 10:22 PM, Madabhattula Rajesh Kumar <
> mrajaforu@gmail.com> wrote:
>
>> Hello Team,
>>
>> I'm new to flume. Could you please help me on below query
>>
>> 1. How to configure JMS source(TIBCO and ActiveMQ) in flume?
>> 2. Do i need to write custom source for above requirement? (Or) Out
>> Of the Box flume JMS source implementation is fine?
>> 3. What is the best channel mechanism for above requirement?
>> 4. Each message will be sink into HDFS? (or) Can we consolidate some
>> messages and sink into HDFS?
>>
>> Thank you in advance
>>
>> Regards,
>> Rajesh
>>
>
>
Re: Flume JMS source
Posted by richard ross <ri...@gmail.com>.
Hello Rajesh:
I just worked on a POC for the work that I am doing with JMS --> HDFS
configuration using ActiveMQ. I was able to use everything right out of the
box, going with JMS Source --> Memory Channel --> HDFS Sink, based on the
guide (https://flume.apache.org/FlumeUserGuide.html). Below is my example
config that I was playing around with. Note that for ActiveMQ, you do not
want to specify the connectionFactory in the source configuration (
https://flume.apache.org/FlumeUserGuide.html#jms-source). A documentation
defect was created for this point (
https://issues.apache.org/jira/browse/FLUME-2339). As for questions 3 and
4, I will let someone with more experience answer.
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe the source
#a1.sources.r1.type = netcat
#a1.sources.r1.bind = localhost
#a1.sources.r1.port = 44444
a1.sources.r1.type = jms
a1.sources.r1.channels = c1
a1.sources.r1.initialContextFactory =
org.apache.activemq.jndi.ActiveMQInitialContextFactory
a1.sources.r1.providerURL = tcp://<activemqserver>:61616
a1.sources.r1.destinationName = TESTQUEUE
a1.sources.r1.destinationType = QUEUE
# Describe the sink
#a1.sinks.k1.type = logger
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://<namenode>:8020/user/root/flumetest/%Y%m%d
a1.sinks.k1.hdfs.filePrefix = messages
a1.sinks.k1.hdfs.rollInterval = 30
#a1.sinks.k1.hdfs.rollSize = 1024
a1.sinks.k1.hdfs.rollCount = 1000
a1.sinks.k1.hdfs.fileType = DataStream
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
Hope this helps,
Richard.
On Mon, Mar 3, 2014 at 10:22 PM, Madabhattula Rajesh Kumar <
mrajaforu@gmail.com> wrote:
> Hello Team,
>
> I'm new to flume. Could you please help me on below query
>
> 1. How to configure JMS source(TIBCO and ActiveMQ) in flume?
> 2. Do i need to write custom source for above requirement? (Or) Out Of
> the Box flume JMS source implementation is fine?
> 3. What is the best channel mechanism for above requirement?
> 4. Each message will be sink into HDFS? (or) Can we consolidate some
> messages and sink into HDFS?
>
> Thank you in advance
>
> Regards,
> Rajesh
>