You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "David Handermann (Jira)" <ji...@apache.org> on 2024/10/28 14:45:00 UTC

[jira] [Updated] (NIFI-13544) Set Thread Executors to Concurrent Tasks configured in PublishGCPubSub

     [ https://issues.apache.org/jira/browse/NIFI-13544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Handermann updated NIFI-13544:
------------------------------------
    Summary: Set Thread Executors to Concurrent Tasks configured in PublishGCPubSub  (was: Add property to set the size of the threadpool of pubsub publish)

> Set Thread Executors to Concurrent Tasks configured in PublishGCPubSub
> ----------------------------------------------------------------------
>
>                 Key: NIFI-13544
>                 URL: https://issues.apache.org/jira/browse/NIFI-13544
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 1.27.0
>            Reporter: Paul-Adrien Cordonnier
>            Priority: Major
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> The GCP java-pubsub library has a default behaviour of creating a threadpool of 5 threads per  processor on the machine. On a typical production server it can be 200 threads per processor, mutliplied by the number of GCP publish processor it can very well overwhelm a JVM.
> [Link to the gcp library snippet|https://github.com/googleapis/java-pubsub/blob/4e17c34880d0498d2bd891ea7caa8fe095054016/google-cloud-pubsub/src/main/java/com/google/cloud/pubsub/v1/Publisher.java#L712-L716]
> Similar to whats its done with [putkudu|https://github.com/apache/nifi/blob/support/nifi-1.x/nifi-nar-bundles/nifi-kudu-bundle/nifi-kudu-processors/src/main/java/org/apache/nifi/processors/kudu/AbstractKuduProcessor.java#L148]. I think we should be able to set the size of the threadpool as a property of the processor. Default behaviour being the one in pubsub-library so that there are no impact on performance after upgrading.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)