Re: Contribution of a new tailing source

[Juhani Connolly <ju...@cyberagent.co.jp>]

Hey all,

I've not been active on flume lately as my colleague Iijima-san here has 
taken over most of our internal flume related work while I've moved on 
to other projects.

This new tail source is a java version of an internal program we've been 
using for over an year(written in python but that had several 
idiosyncrasies intended for our use which made it not particularly 
useful to be contributed). We've had this new source in production for a 
month now and it was in testing for a long time before that. To my 
knowledge we have not lost any logs when using it(and an appropriate 
pipeline of course), though I believe there may be some extreme corner 
cases involving either very fast rotation of log files or moving log 
files intended to be read to another directory before they've been 
opened(both should never happen in any kind of normal environment).

As was mentioned, it uses inodes to detect file rotations and is thus 
only suitable for use with unix operating systems. That being said it 
would be good to get this into flume since I think a reliable tailing 
source is something a lot of people have asked for in the past.

Please let us know what you think,

Thanks,
   Juhani

On 10/09/2014 03:22 PM, 飯島賢志 wrote:
> Hello,
>
> We would like to contribute a new tailing source to the Apache Flume
> community.
> We have operated it in production for 1 month and there is no technical
> problem for now.
>
> This source is below:
> ---
> Taildir source watches the specified files, and tails them in nearly
> real-time once appends are detected to these files.
>
> This source is reliable and will not miss data even when the tailing files
> rotate.
> It periodically writes the last read position, inode and absolute path of
> each file in a position file using the JSON format.
> ---
>
> If Flume is stopped or down for some reason, it can restart tailing from
> the position written in the position file.
>
> This source is very convenient for tailing files and recovering form the
> system failures such as the Flume process dying..
> But it requires Unix-style file system and Java 1.7 or later because it
> uses inode to identify files.
> Can we contribute it to the Apache Flume community?
> If acceptable I will attach the patch of it in JIRA.
>
> Thank you,
> -Satoshi iijima
>