[jira] [Created] (NIFI-10073) Adding new load balancing strategy for better utilisation and to avoid backpressure if possible

["Simon Bence (Jira)" <ji...@apache.org>]

Simon Bence created NIFI-10073:
----------------------------------

             Summary: Adding new load balancing strategy for better utilisation and to avoid backpressure if possible
                 Key: NIFI-10073
                 URL: https://issues.apache.org/jira/browse/NIFI-10073
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Core Framework
            Reporter: Simon Bence
            Assignee: Simon Bence


With the current load balancing for connections, NiFi might end up in a situation where the utilisation of the nodes differ highly. In extreme cases backpressure might be applied even if  there are nodes might accept data.

In order to resolve this, a new load balancing strategy needs to be introduced with the following characteristics:
- Work stealing should be possible, thus the strategy should try to keep all nodes busy.
- If any node can accept data we should not apply backpressure

To reach this, when the newly introduced strategi is being applied, a shared partition will be used by the remote partitons. When the flow files are sent to a remote node, the decision must be made, how many of them should be sent. Helping this decision the load balancing protocol needs to be modified in the way, instead of sending back SPACE_AVAILABLE and QUEUE_FULL, the nodes should provide a more detailed information.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)