Fwd: Issue : Flume performance

[ajeet singh <aj...@gmail.com>]

---------- Forwarded message ----------
From: ajeet singh <aj...@gmail.com>
Date: Mon, Apr 6, 2015 at 12:03 AM
Subject: Issue : Flume performance
To: user@flume.apache.org


I am using flume for logging events from aws servers to KAFKA. I am using
logback as flume source, and kafka-sink as sink. In servers we are getting
3000 events/ per seconds, and there are almost 30 servers running and one
central kafka. At each server flume process is running which will be
writing to central kafka.
             I am getting some strange situations like at some servers
flume is able for process the event with rate in which it's getting  the
event but on some servers some time it get slow ( like it's getting event
with rate 3k / sec but transferring events to kafka with rate 1Kv per sec).
This way 2K /per sec events are getting added to flume channel, and after
3-4 hours channel overflow starts. I compared both type of servers where
flume is working fine, and where flume is slow, all parameters ( load, cpu
utilization, memory usages and disk writes) are almost same.  What can be
reason for this strange behavior ??

Thanks
Ajeet



-- 
AJEET PRATAP SINGH

Re: Issue : Flume performance

[Ashish <pa...@gmail.com>]

Missed this thread. No direct solution comes to my mind for this

Couple of things, since the degradation is occurring only with few instances.

1. Find out what is different on the machines seeing degradation from
the others (including OS configurations)
2. Is it the degradation random or occurring on specific hosts

Put me in this situation and I would do following

1. Try to look out for a Pattern of degradation, similar hosts, some
event storm or could be anything
2. If you have monitoring system, see what's happening around during
the degradation, heavy GC activity.

Once you get a similar pattern, perhaps debugging and solving it would
be easier.

On Mon, Apr 6, 2015 at 12:20 AM, ajeet singh <aj...@gmail.com> wrote:
> ---------- Forwarded message ----------
> From: ajeet singh <aj...@gmail.com>
> Date: Mon, Apr 6, 2015 at 12:03 AM
> Subject: Issue : Flume performance
> To: user@flume.apache.org
>
>
> I am using flume for logging events from aws servers to KAFKA. I am using
> logback as flume source, and kafka-sink as sink. In servers we are getting
> 3000 events/ per seconds, and there are almost 30 servers running and one
> central kafka. At each server flume process is running which will be
> writing to central kafka.
>              I am getting some strange situations like at some servers
> flume is able for process the event with rate in which it's getting  the
> event but on some servers some time it get slow ( like it's getting event
> with rate 3k / sec but transferring events to kafka with rate 1Kv per sec).
> This way 2K /per sec events are getting added to flume channel, and after
> 3-4 hours channel overflow starts. I compared both type of servers where
> flume is working fine, and where flume is slow, all parameters ( load, cpu
> utilization, memory usages and disk writes) are almost same.  What can be
> reason for this strange behavior ??
>
> Thanks
> Ajeet
>
>
>
> --
> AJEET PRATAP SINGH



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal