RE: Scale of a flume collector

[Chalcy Raja <Ch...@careerbuilder.com>]

Hi Kim,

How are you handling realtime collection?  I am okay with a few minutes delay, but the collector is collecting only once every hour.  What configuration I should use to set the collector to collect every minute?

Have you set up the flume yet?  How are you handling agent dying?

Thanks,
Chalcy
________________________________________
From: Kim, Jongkook  [jongkook.kim@citi.com]
Sent: Tuesday, February 14, 2012 2:25 PM
To: 'flume-user@incubator.apache.org'
Subject: Scale of a flume collector

Hi all.

I'm in the middle of hardware provisioning for flume-hbase-hadoop solution.
The plan is that flume agents collect and pass log data to collectors and the collectors write data into hbase using sink.
The question is a flume collector's scale.

Flume agents:250
Data receiving ratio: 5.78MB/second
Data writing ratio: 17.9MB/second
Number of data nodes: 12

This system will be used to provide real-time use case, so there shouldn't be delay.
How many collectors required to handle this request?

Thanks in advance,