You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Roshan Naik (JIRA)" <ji...@apache.org> on 2015/08/05 12:21:05 UTC
[jira] [Updated] (FLUME-2498) Implement Taildir Source
[ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roshan Naik updated FLUME-2498:
-------------------------------
Attachment: FLUME-2498-3.patch
Given that this a useful patch that has been waiting for sometime .. i will try to fast track it. So I am uploading a revised patch with the below noted fixes. I will make my review comments in the next comment:
*Build changes:*
- fixed merge issues on to 1.7 latest
- changed version to 1.7.0-SNAPSHOT
- added taildir-source to flume-ng-dist/pom.xml
- fixed : flume-taildir-source/pom.xml had wrong groupid
*Code/doc changes:*
- provided implementation for abstract methods : getBackOffSleepIncrement & getMaxBackOffSleepInterval
- documented both these two new settings
- provided support for short name (instead of FQCN) : sources.s1.type = TAILDIR
- documented the name of the header used by the byteOffsetHeader setting
- typos in error message
> Implement Taildir Source
> ------------------------
>
> Key: FLUME-2498
> URL: https://issues.apache.org/jira/browse/FLUME-2498
> Project: Flume
> Issue Type: New Feature
> Components: Sinks+Sources
> Reporter: Satoshi Iijima
> Fix For: v1.7.0
>
> Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, FLUME-2498.patch
>
>
> This is the proposal of implementing a new tailing source.
> This source watches the specified files, and tails them in nearly real-time once appends are detected to these files.
> * This source is reliable and will not miss data even when the tailing files rotate.
> * It periodically writes the last read position of each file in a position file using the JSON format.
> * If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file.
> * It can add event headers to each tailing file group.
> A attached patch includes a config documentation of this.
> This source requires Unix-style file system and Java 1.7 or later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)