You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Chhaya Vishwakarma <Ch...@lntinfotech.com> on 2014/03/06 05:27:07 UTC
Multiple flume agent on single machine
Hi
I have a problem statement where i need to collect data from multiple machines(each machine forwarding data for different format) to centralized storage(Hbase).
I need to make a different table for each type of log collected.
How can i do it using flume? Is it possible to run multiple flume agent on same machine ? I can assign each type to one agent which will write to hbase.
Regards,
Chhaya Vishwakarma
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"
Re: Multiple flume agent on single machine
Posted by Christopher Shannon <cs...@gmail.com>.
This looks to be just what I need.
Thanx, Jeff
- Chris
On Mar 13, 2014 4:52 PM, "Jeff Lord" <jl...@cloudera.com> wrote:
> You can setup flume to use hdfs.proxyUser
>
>
> https://cwiki.apache.org/confluence/display/FLUME/Flume+1.x+Secure+HDFS+Setup
>
>
> On Thu, Mar 13, 2014 at 2:26 PM, Christopher Shannon <
> cshannon108@gmail.com> wrote:
>
>> What if your sinks have to write out to destinations that have different
>> users and different levels of authorization? How is security managed by
>> Flume?
>> On Mar 6, 2014 8:56 AM, "Simeon Zaharici" <si...@yahoo.com>
>> wrote:
>>
>>> Hello
>>>
>>> You do not need to run multiple flume agents, although this is possible.
>>> You could just define multiple sources running on different ports in the
>>> same flume agent, each one writing to its own Hbase sink.
>>>
>>> Regards,
>>> Simeon
>>>
>>>
>>>
>>> On Wednesday, March 5, 2014 11:30:56 PM, Chhaya Vishwakarma <
>>> Chhaya.Vishwakarma@lntinfotech.com> wrote:
>>> Hi
>>>
>>> I have a problem statement where i need to collect data from multiple
>>> machines(each machine forwarding data for different format) to centralized
>>> storage(Hbase).
>>> I need to make a different table for each type of log collected.
>>> How can i do it using flume? Is it possible to run multiple flume agent
>>> on same machine ? I can assign each type to one agent which will write to
>>> hbase.
>>>
>>>
>>> Regards,
>>> Chhaya Vishwakarma
>>>
>>>
>>> ------------------------------
>>> The contents of this e-mail and any attachment(s) may contain
>>> confidential or privileged information for the intended recipient(s).
>>> Unintended recipients are prohibited from taking action on the basis of
>>> information in this e-mail and using or disseminating the information, and
>>> must notify the sender and delete it from their system. L&T Infotech will
>>> not accept responsibility or liability for the accuracy or completeness of,
>>> or the presence of any virus or disabling code in this e-mail"
>>>
>>>
>>>
>
Re: Multiple flume agent on single machine
Posted by Jeff Lord <jl...@cloudera.com>.
You can setup flume to use hdfs.proxyUser
https://cwiki.apache.org/confluence/display/FLUME/Flume+1.x+Secure+HDFS+Setup
On Thu, Mar 13, 2014 at 2:26 PM, Christopher Shannon
<cs...@gmail.com>wrote:
> What if your sinks have to write out to destinations that have different
> users and different levels of authorization? How is security managed by
> Flume?
> On Mar 6, 2014 8:56 AM, "Simeon Zaharici" <si...@yahoo.com>
> wrote:
>
>> Hello
>>
>> You do not need to run multiple flume agents, although this is possible.
>> You could just define multiple sources running on different ports in the
>> same flume agent, each one writing to its own Hbase sink.
>>
>> Regards,
>> Simeon
>>
>>
>>
>> On Wednesday, March 5, 2014 11:30:56 PM, Chhaya Vishwakarma <
>> Chhaya.Vishwakarma@lntinfotech.com> wrote:
>> Hi
>>
>> I have a problem statement where i need to collect data from multiple
>> machines(each machine forwarding data for different format) to centralized
>> storage(Hbase).
>> I need to make a different table for each type of log collected.
>> How can i do it using flume? Is it possible to run multiple flume agent
>> on same machine ? I can assign each type to one agent which will write to
>> hbase.
>>
>>
>> Regards,
>> Chhaya Vishwakarma
>>
>>
>> ------------------------------
>> The contents of this e-mail and any attachment(s) may contain
>> confidential or privileged information for the intended recipient(s).
>> Unintended recipients are prohibited from taking action on the basis of
>> information in this e-mail and using or disseminating the information, and
>> must notify the sender and delete it from their system. L&T Infotech will
>> not accept responsibility or liability for the accuracy or completeness of,
>> or the presence of any virus or disabling code in this e-mail"
>>
>>
>>
Re: Multiple flume agent on single machine
Posted by Christopher Shannon <cs...@gmail.com>.
What if your sinks have to write out to destinations that have different
users and different levels of authorization? How is security managed by
Flume?
On Mar 6, 2014 8:56 AM, "Simeon Zaharici" <si...@yahoo.com> wrote:
> Hello
>
> You do not need to run multiple flume agents, although this is possible.
> You could just define multiple sources running on different ports in the
> same flume agent, each one writing to its own Hbase sink.
>
> Regards,
> Simeon
>
>
>
> On Wednesday, March 5, 2014 11:30:56 PM, Chhaya Vishwakarma <
> Chhaya.Vishwakarma@lntinfotech.com> wrote:
> Hi
>
> I have a problem statement where i need to collect data from multiple
> machines(each machine forwarding data for different format) to centralized
> storage(Hbase).
> I need to make a different table for each type of log collected.
> How can i do it using flume? Is it possible to run multiple flume agent on
> same machine ? I can assign each type to one agent which will write to
> hbase.
>
>
> Regards,
> Chhaya Vishwakarma
>
>
> ------------------------------
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>
>
>
Re: Multiple flume agent on single machine
Posted by Simeon Zaharici <si...@yahoo.com>.
Here is a simple example of 2 netcat sources sending to 2 different elasticsearch clusters. The same principle would apply for HBase as well
#Flow definition
agent.sources = netcat anothernetcat
agent.channels = netcat-channel-es anothernetcat-channel-es
agent.sinks = netcat-sink-es anothernetcat-sink-es
# For each one of the sources, the type is defined
agent.sources.netcat.type = netcat
agent.sources.netcat.bind = 0.0.0.0
agent.sources.netcat.port = 5150
agent.sources.anothernetcat.type = netcat
agent.sources.anothernetcat.bind = 0.0.0.0
agent.sources.anothernetcat.port = 5155
# The channels can be defined as follows.
agent.sources.netcat.channels = netcat-channel-es
agent.sources.anothernetcat.channels = anothernetcat-channel-es
# Each sink's type must be defined
agent.sinks.netcat-sink-es.type = elasticsearch
agent.sinks.netcat-sink-es.hostNames = server1:9300,server2:9300,server3:9300
agent.sinks.netcat-sink-es.indexName = flume_netcat_index
agent.sinks.netcat-sink-es.indexType = bar_type
agent.sinks.netcat-sink-es.clusterName = netcat
agent.sinks.netcat-sink-es.batchSize = 500
agent.sinks.netcat-sink-es.ttl = 5
agent.sinks.anothernetcat-sink-es.type = elasticsearch
agent.sinks.anothernetcat-sink-es.hostNames = server4:9300,server5:9300,server6:9300
agent.sinks.anothernetcat-sink-es.indexName = flume_anothernetcat_index
agent.sinks.anothernetcat-sink-es.indexType = bar_type
agent.sinks.anothernetcat-sink-es.clusterName = anothernetcat
agent.sinks.anothernetcat-sink-es.batchSize = 500
agent.sinks.anothernetcat-sink-es.ttl = 5
#Specify the channel the sink should use
agent.sinks.netcat-sink-es.channel = netcat-channel-es
agent.sinks.anothernetcat-sink-es.channel = anothernetcat-channel-es
# Each channel's type is defined.
agent.channels.netcat-channel-es.type = memory
agent.channels.anothernetcat-channel-es.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
agent.channels.netcat-channel-es.capacity = 1000000
agent.channels.anothernetcat-channel-es.capacity = 1000000
agent.channels.netcat-channel-es.transactionCapacity = 1000
agent.channels.anothernetcat-channel-es.transactionCapacity = 1000
On Thursday, March 6, 2014 11:03:08 PM, Chhaya Vishwakarma <Ch...@lntinfotech.com> wrote:
Thank you for your reply do you have any sample for the same?
From:Simeon Zaharici [mailto:simeon_zaharici@yahoo.com]
Sent: Thursday, March 06, 2014 8:23 PM
To: user@flume.apache.org
Subject: Re: Multiple flume agent on single machine
Hello
You do not need to run multiple flume agents, although this is possible.
You could just define multiple sources running on different ports in the same flume agent, each one writing to its own Hbase sink.
Regards,
Simeon
On Wednesday, March 5, 2014 11:30:56 PM, Chhaya Vishwakarma <Ch...@lntinfotech.com> wrote:
Hi
I have a problem statement where i need to collect data from multiple machines(each machine forwarding data for different format) to centralized storage(Hbase).
I need to make a different table for each type of log collected.
How can i do it using flume? Is it possible to run multiple flume agent on same machine ? I can assign each type to one agent which will write to hbase.
Regards,
Chhaya Vishwakarma
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"
RE: Multiple flume agent on single machine
Posted by Chhaya Vishwakarma <Ch...@lntinfotech.com>.
Thank you for your reply do you have any sample for the same?
From: Simeon Zaharici [mailto:simeon_zaharici@yahoo.com]
Sent: Thursday, March 06, 2014 8:23 PM
To: user@flume.apache.org
Subject: Re: Multiple flume agent on single machine
Hello
You do not need to run multiple flume agents, although this is possible.
You could just define multiple sources running on different ports in the same flume agent, each one writing to its own Hbase sink.
Regards,
Simeon
On Wednesday, March 5, 2014 11:30:56 PM, Chhaya Vishwakarma <Ch...@lntinfotech.com>> wrote:
Hi
I have a problem statement where i need to collect data from multiple machines(each machine forwarding data for different format) to centralized storage(Hbase).
I need to make a different table for each type of log collected.
How can i do it using flume? Is it possible to run multiple flume agent on same machine ? I can assign each type to one agent which will write to hbase.
Regards,
Chhaya Vishwakarma
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"
Re: Multiple flume agent on single machine
Posted by Simeon Zaharici <si...@yahoo.com>.
Hello
You do not need to run multiple flume agents, although this is possible.
You could just define multiple sources running on different ports in the same flume agent, each one writing to its own Hbase sink.
Regards,
Simeon
On Wednesday, March 5, 2014 11:30:56 PM, Chhaya Vishwakarma <Ch...@lntinfotech.com> wrote:
Hi
I have a problem statement where i need to collect data from multiple machines(each machine forwarding data for different format) to centralized storage(Hbase).
I need to make a different table for each type of log collected.
How can i do it using flume? Is it possible to run multiple flume agent on same machine ? I can assign each type to one agent which will write to hbase.
Regards,
Chhaya Vishwakarma
________________________________
The contents of this e-mail and any attachment(s) may contain confidential or privileged information for the intended recipient(s). Unintended recipients are prohibited from taking action on the basis of information in this e-mail and using or disseminating the information, and must notify the sender and delete it from their system. L&T Infotech will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in this e-mail"