You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Cameron Gandevia <cg...@gmail.com> on 2012/02/23 19:23:48 UTC

Tail Rolling Files

Hi

We use the TailSource in flume to tail our log files. Whenever the log
files reach 1gb in size they are rolled. We are experiencing a problem
where flume cannot keep up with the log write rate and eventually missing
an entire file because it rolls over before flume notices. I think this
happens because flume doesn't detect the roll until it finishes tailing the
current file. I am think a simple solution to start would be to include a
timestamp in the log files and use a tailDir so that every file is seen. My
only concern is we will end up with more and more tail threads as the
writing gets further and further ahead.

Just wondering if anyone else experienced and dealt with this problem?

Thanks

Re: Tail Rolling Files

Posted by alo alt <wg...@googlemail.com>.
Hi,

pls send the config for your sink.
You create one file in a directory, and after that file has reached 1GB you reset that? 

best,
Alex

--
Alexander Lorenz
http://mapredit.blogspot.com

On Feb 23, 2012, at 10:23 AM, Cameron Gandevia wrote:

> Hi
> 
> We use the TailSource in flume to tail our log files. Whenever the log files reach 1gb in size they are rolled. We are experiencing a problem where flume cannot keep up with the log write rate and eventually missing an entire file because it rolls over before flume notices. I think this happens because flume doesn't detect the roll until it finishes tailing the current file. I am think a simple solution to start would be to include a timestamp in the log files and use a tailDir so that every file is seen. My only concern is we will end up with more and more tail threads as the writing gets further and further ahead.
> 
> Just wondering if anyone else experienced and dealt with this problem?
> 
> Thanks
> 
>

Re: Tail Rolling Files

Posted by alo alt <wg...@googlemail.com>.
Hi,

pls send the config for your sink.
You create one file in a directory, and after that file has reached 1GB you reset that? 

best,
 Alex

--
Alexander Lorenz
http://mapredit.blogspot.com

On Feb 23, 2012, at 10:23 AM, Cameron Gandevia wrote:

> Hi
> 
> We use the TailSource in flume to tail our log files. Whenever the log files reach 1gb in size they are rolled. We are experiencing a problem where flume cannot keep up with the log write rate and eventually missing an entire file because it rolls over before flume notices. I think this happens because flume doesn't detect the roll until it finishes tailing the current file. I am think a simple solution to start would be to include a timestamp in the log files and use a tailDir so that every file is seen. My only concern is we will end up with more and more tail threads as the writing gets further and further ahead.
> 
> Just wondering if anyone else experienced and dealt with this problem?
> 
> Thanks
> 
>