Does Flume need a workload generator?

[Patrick Wendell <pw...@gmail.com>]

Hi All,

I was trying to debug a memory leak and it doesn't seem like there is
a good way to generate workload for Flume at a configurable rate, or
with configurable characteristics like payload size and header size.
The sequence source is something like this, but you can't control the
rate of event generation at all.

What do people do now when benchmarking or debugging Flume? I was
thinking of writing a source that generates event streams with
configurable properties. Would that be useful? Or is this need already
met somehow else?

- Patrick

Re: Does Flume need a workload generator?

[Mike Percy <mp...@apache.org>]

Patrick,
Will and I have put together a couple of classes that do much (not all) of
what you're describing for a Syslog TCP client and an Avro RPC client. I've
been meaning to commit them to Flume SVN but I haven't gotten around to it
yet. Here is the syslog one:
https://github.com/mpercy/flume-load-gen/blob/master/src/main/java/org/apache/flume/tcphammer/Hammer.java

I can give you an Avro RPC one which is similar.

Regards,
Mike

On Mon, Jul 16, 2012 at 9:28 PM, Patrick Wendell <pw...@gmail.com> wrote:

> Hi All,
>
> I was trying to debug a memory leak and it doesn't seem like there is
> a good way to generate workload for Flume at a configurable rate, or
> with configurable characteristics like payload size and header size.
> The sequence source is something like this, but you can't control the
> rate of event generation at all.
>
> What do people do now when benchmarking or debugging Flume? I was
> thinking of writing a source that generates event streams with
> configurable properties. Would that be useful? Or is this need already
> met somehow else?
>
> - Patrick
>

Re: Does Flume need a workload generator?

[Otis Gospodnetic <ot...@yahoo.com>]

Hi,

Have a look at ActionGenerator we put together for Solr & ElasticSearch so far - it has the framework that's extensible and that allows one to easily add support for Flume, for example.

https://github.com/sematext/ActionGenerator 


A pull request with code that can push stuff in Flume would be great.

Otis 
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 



>________________________________
> From: Patrick Wendell <pw...@gmail.com>
>To: flume-dev@incubator.apache.org 
>Sent: Tuesday, July 17, 2012 12:28 AM
>Subject: Does Flume need a workload generator?
> 
>Hi All,
>
>I was trying to debug a memory leak and it doesn't seem like there is
>a good way to generate workload for Flume at a configurable rate, or
>with configurable characteristics like payload size and header size.
>The sequence source is something like this, but you can't control the
>rate of event generation at all.
>
>What do people do now when benchmarking or debugging Flume? I was
>thinking of writing a source that generates event streams with
>configurable properties. Would that be useful? Or is this need already
>met somehow else?
>
>- Patrick
>
>
>