[jira] [Commented] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value

["David Handermann (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509900#comment-17509900 ] 

David Handermann commented on NIFI-9820:
----------------------------------------

Thanks for raising this issue [~jzahner]. The current default value derives from the internal default value used within the Kudu Client library. As you have noted, however, configuring multiple PutKudu Processors results in memory usage problems.

Some NiFi flow configurations can benefit from a larger number, where only a couple PutKudu Processors may be configured. Setting the property based on the number of concurrent tasks is a good approach in general.

Changing the default value to the number of reported CPU cores is one simple way forward.  Another option could be setting the default value to half the number of CPU cores, with a minimum value of 1. I am inclined to go with the first option, and I can put forward a pull request for further consideration.

> Change PutKudu Property "Kudu Client Worker Count" Default Value
> ----------------------------------------------------------------
>
>                 Key: NIFI-9820
>                 URL: https://issues.apache.org/jira/browse/NIFI-9820
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 1.15.3
>            Reporter: Josef Zahner
>            Assignee: David Handermann
>            Priority: Minor
>
> The PutKudu processor property "Kudu Client Worker Count" has a suboptimal value. Please don't use the current "number of CPUs multiplied by 2" behaviour as it leads to a massive amount of workers in our case with physical servers. We have a 8-node cluster where each server has 64 CPUs. We have about 30 PutKudu processors configured -> a lot of worker threads per default just for kudu.
> We have changed the number of worker threads in our case to the number of concurrent tasks. I don't know, maybe it would be great to set it a bit higher than that, but to be honest, I don't exactly understand the impact. It looks still fast with the current config.
> *To sum it up, please set a low default value (eg. 4 or 8) for the property "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu processor.*
> Btw. are there any suggestions how big the number should be?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)