[jira] [Commented] (FLUME-2498) Implement Taildir Source

["mouwei (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266010#comment-15266010 ] 

mouwei commented on FLUME-2498:
-------------------------------

Hi,
I fond a bug of this tailsource.

When I use the regular expression to match files under a folder. when some of file was rolling by log4j, this file's start position which is used to record tail position will be setted to 0.  And then all of matched files will be readed again.

after checking the code. I find below info:
The process() will update all of inodes info by "existingInodes.addAll(reader.updateTailFiles());"
But the the skipToEnd will be setted to "false" when update this file.
" public List<Long> updateTailFiles() throws IOException {
    return updateTailFiles(false);
  }"
when this file was rolled. below code will be executed. this startPos will be setted to 0. It will be readed again.
if (tf == null || !tf.getPath().equals(f.getAbsolutePath())) {
          long startPos = skipToEnd ? f.length() : 0;
          tf = openFile(f, headers, inode, startPos);
}

Does anyone occurred same problem or  is there any setting I missed?




> Implement Taildir Source
> ------------------------
>
>                 Key: FLUME-2498
>                 URL: https://issues.apache.org/jira/browse/FLUME-2498
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>            Reporter: Satoshi Iijima
>             Fix For: v1.7.0
>
>         Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, FLUME-2498-4.patch, FLUME-2498-5.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files rotate.
> * It periodically writes the last read position of each file in a position file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file.
> * It can add event headers to each tailing file group. 
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)