You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Payne (Jira)" <ji...@apache.org> on 2021/07/26 18:30:00 UTC

[jira] [Updated] (NIFI-8773) Allow TailFile to hold off on ingesting lines of text if the full (multi-line) message is not available

     [ https://issues.apache.org/jira/browse/NIFI-8773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Payne updated NIFI-8773:
-----------------------------
    Summary: Allow TailFile to hold off on ingesting lines of text if the full (multi-line) message is not available  (was: Allow TailFile to hold off on ingesting lines of text if the fully message is not available)

> Allow TailFile to hold off on ingesting lines of text if the full (multi-line) message is not available
> -------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-8773
>                 URL: https://issues.apache.org/jira/browse/NIFI-8773
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Extensions
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>             Fix For: 1.15.0
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When using TailFile, there are times when multi-line messages are written to a file. For example, we may have something like:
> {code}
> <1> My Message
> <2> My Message
> <3> My Message
>    A continuation of my message
> {code}
> If TailFile now runs, it will ingest these 4 lines of text as a FlowFile.
> Perhaps the next lines to get written, though, will be something like:
> {code}
>   Another continuation of my message
>   A final continuation
> <4> Another Message
> <5> Yet another Message
> {code}
> And we may want to avoid pulling in lines "<3> My Message" and "   A continuation of my message" until we are able to fully consume the full message.
> We should enable this capability by allowing for a new property that specifies a Regular Expression to run against the start of a line. If we read a line from the file and it matches that Regex, then we know the previous message is complete. Otherwise, the previous message may not be complete and should be buffered (up to some configurable limit, in order to avoid exhausting the Java heap).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)