You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by 吳瑞琳 <rl...@gmail.com> on 2013/02/08 03:36:51 UTC
Flume NG and zookeeper
Hi all,
I am trying to integrate Flume NG and zookeeper. However, I did not find
any configuration about this in Flume NG. Could you please advise how to
deal with this?
Thanks,
RL
Re: Flume NG and zookeeper
Posted by Roshan Naik <ro...@hortonworks.com>.
Flume NG does not need ZK. what are you trying to accomplish ?
-roshan
On Thu, Feb 7, 2013 at 6:36 PM, 吳瑞琳 <rl...@gmail.com> wrote:
> Hi all,
>
> I am trying to integrate Flume NG and zookeeper. However, I did not find
> any configuration about this in Flume NG. Could you please advise how to
> deal with this?
>
> Thanks,
> RL
Re: Flume NG and zookeeper
Posted by Denny Ye <de...@gmail.com>.
Agreed with Roshan, VIP is best choice in present. If you want solution
based on Zookeeper, you should wrap to resolve notification and dynamic
configuration by yourself. Much better if someone can try to post patch for
this, not only load balance RPC client, aren't you?
-Regards
Denny Ye
2013/2/9 Roshan Naik <ro...@hortonworks.com>
> you could put flume behind a VIP and not expose the actual flume nodes
> names to clients.
> -roshan
>
>
> On Fri, Feb 8, 2013 at 6:44 AM, Tech Mail <yn...@gmail.com> wrote:
>
>> Hi,
>>
>> I am looking for similar functionality, but we loose the flexibility.
>>
>> Cannot add a new agent dynamically to the list of collectors.
>> Cannot move the agents from one host to other host or from one datacenter
>> to other.
>> Entire list of agents has to be provided to all end user applications.
>>
>> How to dynamically get the host/port of the agent from the client
>> application?
>> LDAP server server.
>> Any other ways ?
>>
>> Yogi
>>
>> On Feb 7, 2013, at 7:50 PM, ������ <rl...@gmail.com> wrote:
>>
>> Mike,
>>
>> Thanks a lot. It fits our need. I'll try your advice.
>>
>> Regards,
>> RL
>>
>> 2013/2/8 Mike Percy <mp...@apache.org>
>>
>>> Interesting use case. Have you considered using the existing
>>> polling-based load balancing RPC client functionality?
>>>
>>> http://flume.apache.org/FlumeDeveloperGuide.html#loadbalancing-rpc-client
>>>
>>> Regards,
>>> Mike
>>>
>>>
>>> On Thu, Feb 7, 2013 at 7:07 PM, ������ <rl...@gmail.com> wrote:
>>>
>>>> Thank you all.
>>>>
>>>> I am trying configure multiple hosts running Flume to collect data from
>>>> clients. I want to use Zookeeper to track status of Flume hosts so that my
>>>> clients can get the list of available Flume host and send data to one of
>>>> them. I would like to make Flume hosts to register an ephemeral node under
>>>> some Zookeeper node when it is up.
>>>>
>>>> Regards,
>>>> RL
>>>>
>>>>
>>>> 2013/2/8 Mike Percy <mp...@apache.org>
>>>>
>>>>> Integrate in what way?
>>>>>
>>>>>
>>>>> On Thu, Feb 7, 2013 at 6:36 PM, ������ <rl...@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I am trying to integrate Flume NG and zookeeper. However, I did not
>>>>>> find any configuration about this in Flume NG. Could you please advise how
>>>>>> to deal with this?
>>>>>>
>>>>>> Thanks,
>>>>>> RL
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Flume NG and zookeeper
Posted by Roshan Naik <ro...@hortonworks.com>.
you could put flume behind a VIP and not expose the actual flume nodes
names to clients.
-roshan
On Fri, Feb 8, 2013 at 6:44 AM, Tech Mail <yn...@gmail.com> wrote:
> Hi,
>
> I am looking for similar functionality, but we loose the flexibility.
>
> Cannot add a new agent dynamically to the list of collectors.
> Cannot move the agents from one host to other host or from one datacenter
> to other.
> Entire list of agents has to be provided to all end user applications.
>
> How to dynamically get the host/port of the agent from the client
> application?
> LDAP server server.
> Any other ways ?
>
> Yogi
>
> On Feb 7, 2013, at 7:50 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>
> Mike,
>
> Thanks a lot. It fits our need. I'll try your advice.
>
> Regards,
> RL
>
> 2013/2/8 Mike Percy <mp...@apache.org>
>
>> Interesting use case. Have you considered using the existing
>> polling-based load balancing RPC client functionality?
>>
>> http://flume.apache.org/FlumeDeveloperGuide.html#loadbalancing-rpc-client
>>
>> Regards,
>> Mike
>>
>>
>> On Thu, Feb 7, 2013 at 7:07 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>>
>>> Thank you all.
>>>
>>> I am trying configure multiple hosts running Flume to collect data from
>>> clients. I want to use Zookeeper to track status of Flume hosts so that my
>>> clients can get the list of available Flume host and send data to one of
>>> them. I would like to make Flume hosts to register an ephemeral node under
>>> some Zookeeper node when it is up.
>>>
>>> Regards,
>>> RL
>>>
>>>
>>> 2013/2/8 Mike Percy <mp...@apache.org>
>>>
>>>> Integrate in what way?
>>>>
>>>>
>>>> On Thu, Feb 7, 2013 at 6:36 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I am trying to integrate Flume NG and zookeeper. However, I did not
>>>>> find any configuration about this in Flume NG. Could you please advise how
>>>>> to deal with this?
>>>>>
>>>>> Thanks,
>>>>> RL
>>>>
>>>>
>>>>
>>>
>>
>
Re: Flume NG and zookeeper
Posted by Tech Mail <yn...@gmail.com>.
Hi,
I am looking for similar functionality, but we loose the flexibility.
Cannot add a new agent dynamically to the list of collectors.
Cannot move the agents from one host to other host or from one datacenter to other.
Entire list of agents has to be provided to all end user applications.
How to dynamically get the host/port of the agent from the client application?
LDAP server server.
Any other ways ?
Yogi
On Feb 7, 2013, at 7:50 PM, 吳瑞琳 <rl...@gmail.com> wrote:
> Mike,
>
> Thanks a lot. It fits our need. I'll try your advice.
>
> Regards,
> RL
>
> 2013/2/8 Mike Percy <mp...@apache.org>
>> Interesting use case. Have you considered using the existing polling-based load balancing RPC client functionality?
>>
>> http://flume.apache.org/FlumeDeveloperGuide.html#loadbalancing-rpc-client
>>
>> Regards,
>> Mike
>>
>>
>> On Thu, Feb 7, 2013 at 7:07 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>>> Thank you all.
>>>
>>> I am trying configure multiple hosts running Flume to collect data from clients. I want to use Zookeeper to track status of Flume hosts so that my clients can get the list of available Flume host and send data to one of them. I would like to make Flume hosts to register an ephemeral node under some Zookeeper node when it is up.
>>>
>>> Regards,
>>> RL
>>>
>>>
>>> 2013/2/8 Mike Percy <mp...@apache.org>
>>>> Integrate in what way?
>>>>
>>>>
>>>> On Thu, Feb 7, 2013 at 6:36 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>>>>> Hi all,
>>>>>
>>>>> I am trying to integrate Flume NG and zookeeper. However, I did not find any configuration about this in Flume NG. Could you please advise how to deal with this?
>>>>>
>>>>> Thanks,
>>>>> RL
>
Re: Flume NG and zookeeper
Posted by 吳瑞琳 <rl...@gmail.com>.
Mike,
Thanks a lot. It fits our need. I'll try your advice.
Regards,
RL
2013/2/8 Mike Percy <mp...@apache.org>
> Interesting use case. Have you considered using the existing polling-based
> load balancing RPC client functionality?
>
> http://flume.apache.org/FlumeDeveloperGuide.html#loadbalancing-rpc-client
>
> Regards,
> Mike
>
>
> On Thu, Feb 7, 2013 at 7:07 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>
>> Thank you all.
>>
>> I am trying configure multiple hosts running Flume to collect data from
>> clients. I want to use Zookeeper to track status of Flume hosts so that my
>> clients can get the list of available Flume host and send data to one of
>> them. I would like to make Flume hosts to register an ephemeral node under
>> some Zookeeper node when it is up.
>>
>> Regards,
>> RL
>>
>>
>> 2013/2/8 Mike Percy <mp...@apache.org>
>>
>>> Integrate in what way?
>>>
>>>
>>> On Thu, Feb 7, 2013 at 6:36 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I am trying to integrate Flume NG and zookeeper. However, I did not
>>>> find any configuration about this in Flume NG. Could you please advise how
>>>> to deal with this?
>>>>
>>>> Thanks,
>>>> RL
>>>
>>>
>>>
>>
>
Re: Flume NG and zookeeper
Posted by Mike Percy <mp...@apache.org>.
Interesting use case. Have you considered using the existing polling-based
load balancing RPC client functionality?
http://flume.apache.org/FlumeDeveloperGuide.html#loadbalancing-rpc-client
Regards,
Mike
On Thu, Feb 7, 2013 at 7:07 PM, 吳瑞琳 <rl...@gmail.com> wrote:
> Thank you all.
>
> I am trying configure multiple hosts running Flume to collect data from
> clients. I want to use Zookeeper to track status of Flume hosts so that my
> clients can get the list of available Flume host and send data to one of
> them. I would like to make Flume hosts to register an ephemeral node under
> some Zookeeper node when it is up.
>
> Regards,
> RL
>
>
> 2013/2/8 Mike Percy <mp...@apache.org>
>
>> Integrate in what way?
>>
>>
>> On Thu, Feb 7, 2013 at 6:36 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I am trying to integrate Flume NG and zookeeper. However, I did not find
>>> any configuration about this in Flume NG. Could you please advise how to
>>> deal with this?
>>>
>>> Thanks,
>>> RL
>>
>>
>>
>
Re: Flume NG and zookeeper
Posted by 吳瑞琳 <rl...@gmail.com>.
Thank you all.
I am trying configure multiple hosts running Flume to collect data from
clients. I want to use Zookeeper to track status of Flume hosts so that my
clients can get the list of available Flume host and send data to one of
them. I would like to make Flume hosts to register an ephemeral node under
some Zookeeper node when it is up.
Regards,
RL
2013/2/8 Mike Percy <mp...@apache.org>
> Integrate in what way?
>
>
> On Thu, Feb 7, 2013 at 6:36 PM, 吳瑞琳 <rl...@gmail.com> wrote:
>
>> Hi all,
>>
>> I am trying to integrate Flume NG and zookeeper. However, I did not find
>> any configuration about this in Flume NG. Could you please advise how to
>> deal with this?
>>
>> Thanks,
>> RL
>
>
>
Re: Flume NG and zookeeper
Posted by Mike Percy <mp...@apache.org>.
Integrate in what way?
On Thu, Feb 7, 2013 at 6:36 PM, 吳瑞琳 <rl...@gmail.com> wrote:
> Hi all,
>
> I am trying to integrate Flume NG and zookeeper. However, I did not find
> any configuration about this in Flume NG. Could you please advise how to
> deal with this?
>
> Thanks,
> RL
Re: Flume NG and zookeeper
Posted by Brock Noland <br...@cloudera.com>.
NG has no zookeeper integration. What are you trying to do? There is one
open JIRA on this https://issues.apache.org/jira/browse/FLUME-1491
On Thu, Feb 7, 2013 at 8:36 PM, 吳瑞琳 <rl...@gmail.com> wrote:
> Hi all,
>
> I am trying to integrate Flume NG and zookeeper. However, I did not find
> any configuration about this in Flume NG. Could you please advise how to
> deal with this?
>
> Thanks,
> RL
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/