[jira] [Commented] (FLUME-2498) Implement Taildir Source

["sutanu das (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637397#comment-14637397 ] 

sutanu das commented on FLUME-2498:
-----------------------------------

Can this patch be backported for Flume 1.5 please?

We run Hortonworks Flume 1.5.2.2 and they will not backport this patch for us.

Reason we need this patch:

1. We want to restart log-files ingestion as events at a point which flume stopped/recovered, yet, the loglines keep growing/appending

2. We want to get logfile even if logs rotate with new_names eg tail.log.x get rotated to tail.log.y -- where -F of exec source doesnt work and spoolDir doesnt work either (b/c of timestamp limitations)

> Implement Taildir Source
> ------------------------
>
>                 Key: FLUME-2498
>                 URL: https://issues.apache.org/jira/browse/FLUME-2498
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Satoshi Iijima
>             Fix For: v1.7.0
>
>         Attachments: FLUME-2498-2.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files rotate.
> * It periodically writes the last read position of each file in a position file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file.
> * It can add event headers to each tailing file group. 
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)