Configuring flume agents remotely

[yavuz gokirmak <yg...@gmail.com>]

Hi all,

We need to monitor and configure flume agents from a centralized tool. In
previous versions as I know flume-master is used for such kind of
requirements. Flume-master does not exist in flume-ng project.

What is the adviced way for monitoring and configuring flume agents
remotely *

Assume I don't have ssh access to agent machines and therefore I need to
get summary information such as #of events processed, #of files read etc..
from running agents continuously

And change agent configurations dynamically&remotely.

regards

Re: Configuring flume agents remotely

[Jeff Lord <jl...@cloudera.com>]

Monitoring and configuration are two separate things here.
Flume is typically monitored using either ganglia or http/json.
Both methods are documented here:
http://flume.apache.org/FlumeUserGuide.html#monitoring

As for configuration management and changes a common way of handling this
would be to
use a configuration management tool such as chef, puppet or ansible.
The agent polls the conf file every 30 seconds and if changes are detected
than the necessary components will be dynamically reloaded.

Not sure why Kafka is being mentioned as that is an entirely separate
project which has nothing to do with flume.


On Fri, Jan 3, 2014 at 3:19 AM, yavuz gokirmak <yg...@gmail.com> wrote:

> Hi all,
>
> We need to monitor and configure flume agents from a centralized tool. In
> previous versions as I know flume-master is used for such kind of
> requirements. Flume-master does not exist in flume-ng project.
>
> What is the adviced way for monitoring and configuring flume agents
> remotely *
>
> Assume I don't have ssh access to agent machines and therefore I need to
> get summary information such as #of events processed, #of files read etc..
> from running agents continuously
>
> And change agent configurations dynamically&remotely.
>
> regards
>

Re: Configuring flume agents remotely

[Devin Suiter RDX <ds...@rdx.com>]

Not sure if this will fit your situation, but have you looked at Kafka
Messaging? https://kafka.apache.org/

*Devin Suiter*
Jr. Data Solutions Software Engineer
100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212
Google Voice: 412-256-8556 | www.rdx.com


On Fri, Jan 3, 2014 at 6:19 AM, yavuz gokirmak <yg...@gmail.com> wrote:

> Hi all,
>
> We need to monitor and configure flume agents from a centralized tool. In
> previous versions as I know flume-master is used for such kind of
> requirements. Flume-master does not exist in flume-ng project.
>
> What is the adviced way for monitoring and configuring flume agents
> remotely *
>
> Assume I don't have ssh access to agent machines and therefore I need to
> get summary information such as #of events processed, #of files read etc..
> from running agents continuously
>
> And change agent configurations dynamically&remotely.
>
> regards
>