Flume performance tuning

["Ryo.Hongo" <xx...@gmail.com>]

Hi.

I did benchmark of the flume. 3~8ms in one time....
Fluentd is 0.1ms~1ms in one time. So, I want to performance tuning.
Please give me some advise!


[conf/flume-conf.properties]-------
agent.sources = r1
agent.sinks = k1
agent.channels = c1

agent.sources.r1.type = avro
agent.sources.r1.bind = 0.0.0.0
agent.sources.r1.port = 44444

agent.sinks.k1.type = avro
agent.sinks.k1.hostname = xxx.xxx.xxx.xxx
agent.sinks.k1.port = 44444

agent.channels.c1.type = memory
agent.channels.c1.capacity = 1000
agent.channels.c1.transactionCapacity = 100

agent.sources.r1.channels = c1
agent.sinks.k1.channel = c1
---------------------------------------------------

Re: Flume performance tuning

[Brock Noland <br...@cloudera.com>]

Can you share more about your tests? How many events you sent? How those times were calculated, etc.   

--  
Brock Noland
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)


On Wednesday, February 20, 2013 at 9:42 PM, Ryo.Hongo wrote:

> Hi.
>  
> I did benchmark of the flume. 3〜8ms in one time....
> Fluentd is 0.1ms〜1ms in one time. So, I want to performance tuning.
> Please give me some advise!
>  
>  
> [conf/flume-conf.properties]-------
> agent.sources = r1
> agent.sinks = k1
> agent.channels = c1
>  
> agent.sources.r1.type = avro  
> agent.sources.r1.bind = 0.0.0.0
> agent.sources.r1.port = 44444
>  
> agent.sinks.k1.type = avro
> agent.sinks.k1.hostname = xxx.xxx.xxx.xxx
> agent.sinks.k1.port = 44444
>  
> agent.channels.c1.type = memory
> agent.channels.c1.capacity = 1000
> agent.channels.c1.transactionCapacity = 100
>  
> agent.sources.r1.channels = c1
> agent.sinks.k1.channel = c1
>  
> ---------------------------------------------------
>  
>  
>