[jira] [Commented] (NIFI-14753) WriteJsonResult leaves the ByteArrayOutputStream in inconsistent state after trying to write an incompatible schema record

["David Handermann (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18007318#comment-18007318 ] 

David Handermann commented on NIFI-14753:
-----------------------------------------

Thanks for summarizing the current issue and describing some options.

The Context section mentions a ByteArrayOutputStream, is that in reference to a test scenario?

To the substance of the issue, there are several things to consider in terms of expected behavior. Schema inference is an inherently best-effort approach, and will always be subject to problems when newer data does not match older schema definitions. If individual records can vary that significantly, then schema evaluation should be deferred, and consuming from a stream should be limited to preserve the original record content, which is one of the existing configuration options.

If the expected schema is within certain boundaries, then a predefined schema with optional types and fields is one  way to go. The other approach is a much more lenient schema inference approach. Making some adjustments around the edges, such as using {{long}} for any integral number, may help in scenarios like the one described.

Beyond improving schema handling, various types of exceptions should trigger some type of failure behavior. That is the purpose of the parse.failure relationship, so routing to that relationship may be the best option for this scenario. Routing to some new relationship is another approach to consider depending on the Processor.

In any exception case, the solution is not attempting to avoid writing to a FlowFile output stream, but to write applicable records to a failure FlowFile instead.

At this point, I'm not seeing any initial need for changes to the WriteJsonResult class. As mentioned in the discussion for NIFI-14696, other Record Writers have the option to implement different strategies for the level of leniency in schema handling, but the current JSON Record Writer looks like it should retain existing behavior in general.

If there are other similar failure scenarios to the one described, that may also be helpful to consider. In these cases, however, evaluating the larger context of options is the way to go.

> WriteJsonResult leaves the ByteArrayOutputStream in inconsistent state after trying to write an incompatible schema record
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-14753
>                 URL: https://issues.apache.org/jira/browse/NIFI-14753
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>    Affects Versions: 2.4.0
>            Reporter: Dariusz Seweryn
>            Priority: Minor
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> h1. Context
> Consider a WriteJsonResult for schema ["Int Field Name": INT].
> Writing a record with ["Int Field Name": Long.MAX_VALUE] throws an IllegalTypeConversionException — this is expected. Unfortunately the ByteArrayOutputStream (BAOS) is then left with content that is not a valid JSON:
> { "Int Field Name" }
> See [this discussion|https://github.com/apache/nifi/pull/10053#discussion_r2190950596].
> h1. IllegalTypeConversionException handling
> This situation has the following consequences for exception handling apart from handling the offending record in the processor that tried to write the record:
> The same malformed record will most probably surface/cause a warning log in a following processor which can be surprising without the knowledge of this behavior in WriteJsonResult (unless all other processors assume that Malformed Records may appear in consumed FlowFiles and silently drop such records). {_}This can lead to unnecessary work being wasted by people triaging such warning logs{_}.
> h1. Ideal solution
> WriteJsonResult in case of IllegalTypeConversionException does not write any data to the BAOS.
> Such approach would need to buffer all writes to the FlowFile until full record is processed, executing them afterwards. This may impact performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)