You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Kim, Jongkook " <jo...@citi.com> on 2012/10/02 23:02:59 UTC
Flume NG for enterprise environment
Hi,
I'm using flume 0.94 in CDH3 environment and thinking to migrate to CDH4.
The old flume has a master instance to control agents and collectors, so we developed a configuration UI to communicate with a flume master.
>From this UI, user can add, edit, and delete logical nodes.
Does anyone use flume NG on enterprise environment?
Without the master concept, how do you manage all agents and collectors.
If you need to read more files on a machine on flume NG, do you manually update flume.conf file and restart the agent?
Thanks.
RE: Flume NG for enterprise environment
Posted by "Kim, Jongkook " <jo...@citi.com>.
Brock,
Thanks for your response.
We are using Cloudera Manager Free edition for our POC environment, and it doesn't have a flume control.
Dose Enterprise version has it?
Thanks.
-----Original Message-----
From: Brock Noland [mailto:brock@cloudera.com]
Sent: Wednesday, October 03, 2012 2:26 PM
To: Kim, Jongkook [CCC-OT_IT]
Cc: *DE Global E2E Sponge Stakeholders; user@flume.apache.org
Subject: Re: Flume NG for enterprise environment
Hi,
I would use puppet, chef, scp, or Cloudera Manager to manage the configuration files of the flume agents. No sense in writing something custom just to copy files around.
Brock
On Wed, Oct 3, 2012 at 9:39 AM, Kim, Jongkook <jo...@citi.com> wrote:
> Hi Brock,
>
> Without a central place for agent config, how to update flume.conf files on hundreds of agents?
>
> We are trying to give user an ability to update agent setup (previously a logical node) real-time.
>
> About 10 - 30 logical nodes are created per agent and they can be updated or deleted anytime.
>
> Do you have any architectural design of this?
>
>
> Thanks.
> -----Original Message-----
> From: Brock Noland [mailto:brock@cloudera.com]
> Sent: Wednesday, October 03, 2012 10:31 AM
> To: user@flume.apache.org
> Cc: *DE Global E2E Sponge Stakeholders
> Subject: Re: Flume NG for enterprise environment
>
> Hi,
>
> I can confirm that many organizations are using Flume NG in a production enterprise environment. You are correct the concept of a master no longer exists.
>
> Brock
>
> On Tue, Oct 2, 2012 at 4:02 PM, Kim, Jongkook <jo...@citi.com> wrote:
>> Hi,
>>
>>
>>
>> I'm using flume 0.94 in CDH3 environment and thinking to migrate to CDH4.
>>
>>
>>
>> The old flume has a master instance to control agents and collectors,
>> so we developed a configuration UI to communicate with a flume master.
>>
>>
>>
>> From this UI, user can add, edit, and delete logical nodes.
>>
>>
>>
>> Does anyone use flume NG on enterprise environment?
>>
>>
>>
>> Without the master concept, how do you manage all agents and collectors.
>>
>>
>>
>> If you need to read more files on a machine on flume NG, do you
>> manually update flume.conf file and restart the agent?
>>
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Flume NG for enterprise environment
Posted by Brock Noland <br...@cloudera.com>.
Hi,
I would use puppet, chef, scp, or Cloudera Manager to manage the
configuration files of the flume agents. No sense in writing something
custom just to copy files around.
Brock
On Wed, Oct 3, 2012 at 9:39 AM, Kim, Jongkook <jo...@citi.com> wrote:
> Hi Brock,
>
> Without a central place for agent config, how to update flume.conf files on hundreds of agents?
>
> We are trying to give user an ability to update agent setup (previously a logical node) real-time.
>
> About 10 - 30 logical nodes are created per agent and they can be updated or deleted anytime.
>
> Do you have any architectural design of this?
>
>
> Thanks.
> -----Original Message-----
> From: Brock Noland [mailto:brock@cloudera.com]
> Sent: Wednesday, October 03, 2012 10:31 AM
> To: user@flume.apache.org
> Cc: *DE Global E2E Sponge Stakeholders
> Subject: Re: Flume NG for enterprise environment
>
> Hi,
>
> I can confirm that many organizations are using Flume NG in a production enterprise environment. You are correct the concept of a master no longer exists.
>
> Brock
>
> On Tue, Oct 2, 2012 at 4:02 PM, Kim, Jongkook <jo...@citi.com> wrote:
>> Hi,
>>
>>
>>
>> I'm using flume 0.94 in CDH3 environment and thinking to migrate to CDH4.
>>
>>
>>
>> The old flume has a master instance to control agents and collectors,
>> so we developed a configuration UI to communicate with a flume master.
>>
>>
>>
>> From this UI, user can add, edit, and delete logical nodes.
>>
>>
>>
>> Does anyone use flume NG on enterprise environment?
>>
>>
>>
>> Without the master concept, how do you manage all agents and collectors.
>>
>>
>>
>> If you need to read more files on a machine on flume NG, do you
>> manually update flume.conf file and restart the agent?
>>
>>
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
RE: Flume NG for enterprise environment
Posted by "Kim, Jongkook " <jo...@citi.com>.
Hi Brock,
Without a central place for agent config, how to update flume.conf files on hundreds of agents?
We are trying to give user an ability to update agent setup (previously a logical node) real-time.
About 10 - 30 logical nodes are created per agent and they can be updated or deleted anytime.
Do you have any architectural design of this?
Thanks.
-----Original Message-----
From: Brock Noland [mailto:brock@cloudera.com]
Sent: Wednesday, October 03, 2012 10:31 AM
To: user@flume.apache.org
Cc: *DE Global E2E Sponge Stakeholders
Subject: Re: Flume NG for enterprise environment
Hi,
I can confirm that many organizations are using Flume NG in a production enterprise environment. You are correct the concept of a master no longer exists.
Brock
On Tue, Oct 2, 2012 at 4:02 PM, Kim, Jongkook <jo...@citi.com> wrote:
> Hi,
>
>
>
> I'm using flume 0.94 in CDH3 environment and thinking to migrate to CDH4.
>
>
>
> The old flume has a master instance to control agents and collectors,
> so we developed a configuration UI to communicate with a flume master.
>
>
>
> From this UI, user can add, edit, and delete logical nodes.
>
>
>
> Does anyone use flume NG on enterprise environment?
>
>
>
> Without the master concept, how do you manage all agents and collectors.
>
>
>
> If you need to read more files on a machine on flume NG, do you
> manually update flume.conf file and restart the agent?
>
>
>
> Thanks.
>
>
>
>
>
>
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Flume NG for enterprise environment
Posted by Brock Noland <br...@cloudera.com>.
Hi,
I can confirm that many organizations are using Flume NG in a
production enterprise environment. You are correct the concept of a
master no longer exists.
Brock
On Tue, Oct 2, 2012 at 4:02 PM, Kim, Jongkook <jo...@citi.com> wrote:
> Hi,
>
>
>
> I’m using flume 0.94 in CDH3 environment and thinking to migrate to CDH4.
>
>
>
> The old flume has a master instance to control agents and collectors, so we
> developed a configuration UI to communicate with a flume master.
>
>
>
> From this UI, user can add, edit, and delete logical nodes.
>
>
>
> Does anyone use flume NG on enterprise environment?
>
>
>
> Without the master concept, how do you manage all agents and collectors.
>
>
>
> If you need to read more files on a machine on flume NG, do you manually
> update flume.conf file and restart the agent?
>
>
>
> Thanks.
>
>
>
>
>
>
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/