[jira] [Commented] (NIFI-11811) ListS3 Processor doesn't back pressure

["endzeit (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-11811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743402#comment-17743402 ] 

endzeit commented on NIFI-11811:
--------------------------------

By default the back pressure mechanisms of processors only affect the scheduling. That is, if any of the downstream queues is full (according to its configuration), the processor is no longer scheduled until the queue is not full anymore.

However, once the processor is scheduled, it might put as many FlowFiles in the downstream queues as it desires, even if the queue is deemed full based on its back pressure configuration. 

Let's say the downstream queue is limited to 10 FlowFiles and the following processor is not started, thus not removing any FlowFiles from the queue. The ListS3 processor is scheduled (as the queue is not full) and now searches the defined S3 bucket. It might find 50 entries and will create a FlowFile for each of them, even though the queue is set to only take up to 10. Afterwards the queue is full however and the ListS3 processor will not be scheduled.

This is the default behaviour for all processors aligned with the core NiFi framework. Additional work is required in the processor implementation to actively watch for the downstream queues back pressure status. Almost no processor does this, as far as I know. 

It seems like your listing just contains a lot of objects. Unless the processor is scheduled even though the queue is full, this sounds like a feature request rather than a bug to me, e.g. introducing a "Max Batch Size" after which the listing should be stopped. The existing "Listing Batch Size" only controls how many FlowFoles will be commited in a single batch, but does not limit the overall amount of FlowFiles to be emitted during a single scheduled run of the processor.

> ListS3 Processor doesn't back pressure
> --------------------------------------
>
>                 Key: NIFI-11811
>                 URL: https://issues.apache.org/jira/browse/NIFI-11811
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core Framework
>    Affects Versions: 1.16.2
>            Reporter: Daniel Hermon
>            Priority: Critical
>
> Dear NiFi team,
> I have built a data flow for syncing two S3 buckets on two separate sites.
> I am using NiFi because the throughput is much higher than any other tool (aws-s3-cli and RClone)
> The flow is pretty straightforward:
> ListS3 on-site A > FetchS3 from site A  -> FetchS3 from site B (check if file already exists)> (on failure) PutS3Object in site B
> Sometimes site B throws 504 timeout exceptions which starts the back pressure mechanism.
> Everything back pressures perfectly fine but not the ListS3 processor output queue, I have to manually terminate it in order to stop listing new files, even trying to stop it fails. 
> I have tried limiting the size (even though it is 0 bytes flow files since the flow file only contains attributes) and also limiting the number of objects.
>  
> p.s the bucket has 5 Billion objects



--
This message was sent by Atlassian Jira
(v8.20.10#820010)