[jira] [Updated] (NIFI-337) Automated cluster load balancing

["Joseph Witt (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joseph Witt updated NIFI-337:
-----------------------------
    Component/s: Core UI

> Automated cluster load balancing
> --------------------------------
>
>                 Key: NIFI-337
>                 URL: https://issues.apache.org/jira/browse/NIFI-337
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Core Framework, Core UI
>            Reporter: Andrew Purtell
>
> On dev@ in response to an inquiry, from [~joewitt]:
> {quote}
> The processors themselves are available and ready to run on all nodes at all times.  It's really just a question of whether they have data to run on.  We have always taken the view that 'if you want scalable dataflow' use scalable interfaces. And I think that is the way to go in every case you can pull it off. That generally meant one should use datasources which offer queueing semantics where multiple independent nodes can pull from the queue with 'at-least-once' guarantees.  In addition each node has back pressure so if it falls behind it slows its rate of pickup which means other nodes in the cluster can pickup the slack.  This has worked extremely well.
> That said, I recognize that it isn't always possible to use scalable interfaces and given enough non-scalable datasources the cluster could become out of balance.  So this certainly seems like a good / valuable / fun / non-trivial problem to tackle.  If we allow connections between processors to be auto-balanced then it will make for a pretty smooth experience as users won't really have to think too much about it.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)