[jira] [Commented] (NIFI-11341) ListenUDPRecord produces invalid FlowFiles due to Content Repository issues

["Mark Payne (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-11341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706056#comment-17706056 ] 

Mark Payne commented on NIFI-11341:
-----------------------------------

Marking as related to NIFI-10887. NIFI-10887 introduces a NonFlushableOutputStream into the StandardProcessSession. That change did not cause this bug. However, it did expose the bug. Namely, the bug was due to the underlying OutputStream not having an idempotent close() method. Introducing this new NonFlushableOutputStream caused that underlying stream to have its close() method now called twice. This should have been ok but since the close() method was not idempotent, it caused issues.

> ListenUDPRecord produces invalid FlowFiles due to Content Repository issues
> ---------------------------------------------------------------------------
>
>                 Key: NIFI-11341
>                 URL: https://issues.apache.org/jira/browse/NIFI-11341
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>    Affects Versions: 1.20.0
>            Reporter: Peter Kimberley
>            Assignee: Mark Payne
>            Priority: Major
>             Fix For: 2.0.0, 1.21.0
>
>         Attachments: NiFi_Flow.json, image.png
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> In our environment, we use {{ListenUDPRecord}} to collect Syslog messages. This processor is followed by a {{PartitionRecord}} processor that populates an attribute for routing.In release {*}1.19.0{*}, this flow worked without issue. In *1.20.0* though, I am seeing intermittent message truncation in {{{}PartitionRecord{}}}, with bulletin messages like the following appearing regularly:
> {noformat}
> PartitionRecord[id=03ea67a7-0b9c-1c9f-ffff-ffff8d7e5185] Failed to partition FlowFile[filename=ca9c3e11-9365-4ff9-9499-29522fc0cab7]: com.fasterxml.jackson.core.JsonParseException: Unexpected character (',' (code 44)): expected a value
> at [Source: (org.apache.nifi.stream.io.NonCloseableInputStream); line: 1, column: 381]{noformat}
>  
> An example message (note the absence of a Syslog header):
> {noformat}
> itor] [Unit test] Alarm check cfg warning threshold=75 critical threshold=85 warning alarm <...>{noformat}
> {{ListenUDPRecord}} properties are attached.
> h3. Reproduction
> The attached minimal flow illustrates this setup.
>  
> To reproduce this issue, generate improperly-formatted syslog and send to {{ListenUDPRecord}}.
>  
> In my environment, I have two syslog sources feeding this test cluster. Scenario is as follows:
>  # First source (compliant Syslog format) feeds in.
>  # Flow is OK - no bulletins.
>  # Activate second source, which is of an invalid Syslog format and flows to the {{parse.failure}} relationship of {{ListenUDPRecord}}. This is expected - I deal with this gracefully.
>  # Bulletins start firing in {{PartitionRecord}} and the first source starts getting truncated randomly.
> Overall, the majority of messages from the well-formed source make it through. However I'm seeing roughly 1 bulletin every few seconds, which indicates a small proportion of messages are getting truncated.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)