scale of File Tail

[AD <st...@gmail.com>]

hey,

 does anyone know if there are any throughput issues with FileTailing
adaptors?   If i am writing huge files is there risk of missing lines
(especially if the files are getting rotated).  I am just trying to decide
if running map/reduce on a rotated log every X minutes is more safe than
FileTail if the log writes get massively high (thousands/sec)

Cheers,
AD

Re: scale of File Tail

[Eric Yang <er...@gmail.com>]

Huge file is better for file tailing adaptor actually.  FileTailing
adaptor checks every 2 seconds to see if file has rotated.  If the
rotating frequency is high, like rotating every seconds.  Then there
may be a chance of losing track of correct offset in the files.
Rotated log every couple minutes with large size, helps FileTailing
adaptor to track offset in the files correctly.
As far as throughput, it can stream up to 10MB/sec from our performance test.

regards,
Eric

On Thu, Dec 22, 2011 at 6:44 PM, AD <st...@gmail.com> wrote:
> hey,
>
>  does anyone know if there are any throughput issues with FileTailing
> adaptors?   If i am writing huge files is there risk of missing lines
> (especially if the files are getting rotated).  I am just trying to decide
> if running map/reduce on a rotated log every X minutes is more safe than
> FileTail if the log writes get massively high (thousands/sec)
>
> Cheers,
> AD