[jira] [Created] (NIFI-10296) Sometimes "batch.output.XYZ" is missing when Process Group is using batch output mode

["Jan Malcomess (Jira)" <ji...@apache.org>]

Jan Malcomess created NIFI-10296:
------------------------------------

             Summary: Sometimes "batch.output.XYZ" is missing when Process Group is using batch output mode
                 Key: NIFI-10296
                 URL: https://issues.apache.org/jira/browse/NIFI-10296
             Project: Apache NiFi
          Issue Type: Bug
          Components: Core Framework
    Affects Versions: 1.16.0
            Reporter: Jan Malcomess
         Attachments: image-2022-07-29-08-57-03-073.png, image-2022-07-29-09-01-22-133.png, image-2022-07-29-09-01-33-385.png, image-2022-07-29-09-02-46-483.png, image-2022-07-29-09-04-23-684.png, image-2022-07-29-09-05-31-911.png, image-2022-07-29-09-07-57-072.png

I have setup the following flow:

!image-2022-07-29-08-57-03-073.png|width=391,height=294!

The ProcessGroup "Transfer Participation Import Records" is set to batch output mode and has two output ports: ProcessingFailed and ProcessingSucceeded.

I need to set a processing state within the database, depending on whether a failure occured within the Processgroup or not.

Thus, I have configured the following check within the two RouteOnAttribute-Processors connected to the ProcessingSucceded and ProccessingFailed-ports:

The Proccessor "AllProcessingSucceeded?" has
!image-2022-07-29-09-01-33-385.png!

while the Processor "SomeProcessingFailed?" has

!image-2022-07-29-09-02-46-483.png!

Unfortunately, sometimes flow files are emitted from the ProcessGroup without having the "batch.output.XYZ"-attributes set.

For example, I have the following provenance:
!image-2022-07-29-09-04-23-684.png|width=1019,height=172!

Most files have the attributes set correctly, e.g.
!image-2022-07-29-09-05-31-911.png|width=206,height=108!

But the files "e52b2b9c-6078-417b-8095-11c057a2ced1" and "bb37d1a6-a7bf-48a4-ae78-2b2f1239c3e8" (the bottom ones) do not:
!image-2022-07-29-09-07-57-072.png|width=514,height=165!

According to the documentation of the batch output mode for process groups this should never happen and so far, I haven't found a means to reproduce the problem reliably - it just happens (I have included a LogMessage-Processor "LogWarnMessage" so that I can get alerted by monitoring the log files). The error occurs more frequently when the flow is proccessing lots of data (hundreds of thousands of flow files).

Any idea what I should do?

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)