[jira] [Commented] (NIFI-8033) ListS3 processor v1.12.1 does not honor queue thresholds

["Pierre Villard (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238804#comment-17238804 ] 

Pierre Villard commented on NIFI-8033:
--------------------------------------

This is working as designed. Backpressure limits are soft limits. We don't know before hand how many elements are going to be listed and we'll output as many flow files as listed during the operation. If it was working differently before, this was not working the way it should IMO. I think there another JIRA to provide some kind of paging on the ListX processors. Another suggestion is to use the Record Writer property to define a writer so that all the listing info goes into one single flow file and you can then define your flow to process the elements the way you want. This provides much more flexibility and options.

> ListS3 processor v1.12.1 does not honor queue thresholds
> --------------------------------------------------------
>
>                 Key: NIFI-8033
>                 URL: https://issues.apache.org/jira/browse/NIFI-8033
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.12.1
>         Environment: Centos 7, Amazon Cloud, 8 CPU cores, 64 GB RAM
>            Reporter: Dominik Dresel
>            Priority: Major
>
> We are using the ListS3 processor to collect our log data from S3 and process them further.
> With version 1.12.1 (including the fix from NIFI-7874) the processor does not honor the output queue thresholds anymore. I set the flow file count limit to 10, despite of this the List3 processor ingested more than 10.000.000 flow files (and it is still going on).
> If you need any more information I would be happy to provide them.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)