You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/09/30 15:10:23 UTC
[jira] [Commented] (NIFI-2825) Remote Process Group should use num
of queued flow files in remote nodes to distribute load
[ https://issues.apache.org/jira/browse/NIFI-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536241#comment-15536241 ]
ASF GitHub Bot commented on NIFI-2825:
--------------------------------------
GitHub user ijokarumawak opened a pull request:
https://github.com/apache/nifi/pull/1084
NIFI-2825: Fix S2S getPeers flow file count
- Added ClusterWorkload message to retrieve workload information from a
cluster coordinator
- Use cluster workload to return queued flow file count to site-to-site
client so that it can calculate distribution of data transfer
Added related unit tests, as well as test with running NiFi secure/unsecure cluster, and standalone.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijokarumawak/nifi nifi-2825
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/1084.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1084
----
commit 2fde9b9d478f3a421bc883ea0d23a21bfa79d5b2
Author: Koji Kawamura <ij...@apache.org>
Date: 2016-09-28T02:07:22Z
NIFI-2825: Fix S2S getPeers flow file count
- Added ClusterWorkload message to retrieve workload information from a
cluster coordinator
- Use cluster workload to return queued flow file count to site-to-site
client so that it can calculate distribution of data transfer
----
> Remote Process Group should use num of queued flow files in remote nodes to distribute load
> -------------------------------------------------------------------------------------------
>
> Key: NIFI-2825
> URL: https://issues.apache.org/jira/browse/NIFI-2825
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.0.0
> Environment: Clustered
> Reporter: Koji Kawamura
> Assignee: Koji Kawamura
> Fix For: 1.1.0
>
>
> A RPG (Site-to-Site client) should take into account of the number of queued flow files in each remote cluster nodes, to calculate the load distribution balance among target nodes. So that it can send more data to nodes those have less queued files, or receive more data from nodes those have more queued files.
> However, after the clustering mechanism has changed since 1.0, this load distribution logic is not handled appropriately. Site-to-Site remote component always returns flow file count as 0. So RPG distribute load evenly, even if a remote cluster has unbalanced load among nodes.
> We need to fix the remote node API to expose queued flow file count correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)