You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by shekhar sharma <sh...@gmail.com> on 2012/04/13 10:42:53 UTC
Quantifying Flume Parameters
Hello,
I am using Flume trunk (NG), and i am trying to quatify certain parameters
like
1. No of events processed by the flume agent. Can we quantify the
processing of events per source and agent? I mean can i calculate the
number of events processed per second by ve written a source and sink
individually?
2. Also i would like to know how to capture the foot print.
what i am trying to do is,
I have written a python script which is generating 20000 records per
second, and then i have an agent which keeps polling the file and takes the
events and sends it to agent.So i would like to know how many records per
second is processed by the flume? For this i have written a python script
to calculate the throughput, which will poll my final output log file every
six seconds and will calculate how many records are being written every
second.
Is there any other way? it would be of great help...
Regards,
Som
Re: Quantifying Flume Parameters
Posted by alo alt <wg...@googlemail.com>.
Hi,
that depend on the memory channel you configure (Xmx option in flume-env.sh). More memory, more events, more throughput. And the flow depends on the eventsize too.
best,
Alex
--
Alexander Lorenz
http://mapredit.blogspot.com
On Apr 13, 2012, at 10:42 AM, shekhar sharma wrote:
> Hello,
> I am using Flume trunk (NG), and i am trying to quatify certain parameters like
>
> 1. No of events processed by the flume agent. Can we quantify the processing of events per source and agent? I mean can i calculate the number of events processed per second by ve written a source and sink individually?
>
> 2. Also i would like to know how to capture the foot print.
>
> what i am trying to do is,
> I have written a python script which is generating 20000 records per second, and then i have an agent which keeps polling the file and takes the events and sends it to agent.So i would like to know how many records per second is processed by the flume? For this i have written a python script to calculate the throughput, which will poll my final output log file every six seconds and will calculate how many records are being written every second.
>
> Is there any other way? it would be of great help...
>
> Regards,
> Som