[jira] [Comment Edited] (NIFI-9820) Change PutKudu Property "Kudu Client Worker Count" Default Value

["Josef Zahner (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509905#comment-17509905 ] 

Josef Zahner edited comment on NIFI-9820 at 3/21/22, 2:05 PM:
--------------------------------------------------------------

It's really a big advantage that we can change the value with a property since 1.14.0, however I could imagine that the internal default value from the Kudu Client library doesn't expect to have multiple clients (aka processors in nifi) on one node. So it makes sense to use the number of CPUs there. In NiFi however it's a different case, it's very likely that you don't have just one client.

The question is, why should we use a "dynamic" calculated value as any other property (eg. FlowFiles per Batch) is as well just on a fixed value. The user has to test/find anyway a good value in his setup.


was (Author: jzahner):
It's really a big advantage that we can change the value with a property since 1.14.0, however I could imagine that the internal default value from the Kudu Client library doesn't expect to have multiple clients (aka processors in nifi) on one node. So it makes sense to use the number of CPUs there. In NiFi however it's a different case, it's very likely that you don't have just one client.

The question is, why should we use a "dynamic" calculated value as any other property (eg. FlowFiles per Batch) is as well just on a fixed value. The use has to test/find anyway a good value in his setup.

> Change PutKudu Property "Kudu Client Worker Count" Default Value
> ----------------------------------------------------------------
>
>                 Key: NIFI-9820
>                 URL: https://issues.apache.org/jira/browse/NIFI-9820
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 1.15.3
>            Reporter: Josef Zahner
>            Assignee: David Handermann
>            Priority: Minor
>
> The PutKudu processor property "Kudu Client Worker Count" has a suboptimal value. Please don't use the current "number of CPUs multiplied by 2" behaviour as it leads to a massive amount of workers in our case with physical servers. We have a 8-node cluster where each server has 64 CPUs. We have about 30 PutKudu processors configured -> a lot of worker threads per default just for kudu.
> We have changed the number of worker threads in our case to the number of concurrent tasks. I don't know, maybe it would be great to set it a bit higher than that, but to be honest, I don't exactly understand the impact. It looks still fast with the current config.
> *To sum it up, please set a low default value (eg. 4 or 8) for the property "Kudu Client Worker Count" and not a pseudo dynamic one for the PutKudu processor.*
> Btw. are there any suggestions how big the number should be?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)