Re: [jira] [Commented] (NIFI-994) Processor to tail files

[Joe Skora <js...@gmail.com>]

Maybe this processor could copy the behavior of the GNU Coreutils "tail"
command, aka Linux *tail*?  (source is here
<http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f=src/tail.c;h=f916d7460395f0cee52c592bc3d160ac94697e73;hb=HEAD>
)

If it can, it appears to use the Linux iNotify service to track file
changes.  If the files are not local or it encounters iNotify problems, it
falls back to a manual tracking process.  I have never had problems with
that utility, and it can track files across logging restarts, rotation, etc.

On Thu, Sep 24, 2015 at 7:50 PM, Aldrin Piri (JIRA) <ji...@apache.org> wrote:

>
>     [
> https://issues.apache.org/jira/browse/NIFI-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907269#comment-14907269
> ]
>
> Aldrin Piri commented on NIFI-994:
> ----------------------------------
>
> I think we can make a best effort at this, but I don't think there are any
> guarantees that we have all the data.  A lot of this comes down to logging
> provider configuration.  As an example, perhaps there are constraints on
> size or time that cause records to be rotated off.  It could take long
> outages for these environments to develop, but when the data has been
> rolled off, it is gone.  Depending on how markers and such work, this
> brings up some interesting cases to consider when implementing.
>
> As another point of consideration, it would be nice to have a
> property/properties that provide handling for rolling log formats.
> Consider logback and log4j with their date formatted log names.
>
>
>
> > Processor to tail files
> > -----------------------
> >
> >                 Key: NIFI-994
> >                 URL: https://issues.apache.org/jira/browse/NIFI-994
> >             Project: Apache NiFi
> >          Issue Type: New Feature
> >    Affects Versions: 0.4.0
> >            Reporter: Joseph Percivall
> >
> > It's a very common data ingest situation to want to input text into the
> system by "tailing" a file, most commonly log files. Currently we don't
> have an easy way to do this.
> > A simple processor to tail a file would benefit many users. There would
> need to be an option to not just tail a file but pick up where the
> processor left off if it is interrupted.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>