[jira] [Commented] (NIFI-5516) Allow data in a Connection to be Load-Balanced across cluster

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579756#comment-16579756 ] 

ASF GitHub Bot commented on NIFI-5516:
--------------------------------------

GitHub user markap14 opened a pull request:

    https://github.com/apache/nifi/pull/2947

    [WIP] NIFI-5516: Implement Load-Balanced Connections

    This is a very involved new feature. At this point, it is functioning well but has the following caveats:
    - We need to do a better job of reconciling 'client' with nodes in the cluster for authorization purposes.
    - It currently points at a SNAPSHOT version of the nifi-registry. NiFi-Registry will need to be released before this is merged to master. This is due to the VersionedConnection object in the data model being updated such that it now supports configuration of load-balancing properties.
    
    At this point, though, because the changeset is extremely large, I think it makes sense to post a Work-In-Progress PR so that others can begin looking at it. It may also be helpful to reference the Feature Proposal for this feature at https://cwiki.apache.org/confluence/display/NIFI/Load-Balanced+Connections

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/markap14/nifi NIFI-5516

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/2947.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2947
    
----
commit 47addfb1b242d77ca1f147058781be1cd6814fee
Author: Mark Payne <ma...@...>
Date:   2018-06-14T15:57:21Z

    Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue
    Refactored more into AbstractFlowFileQueue
    Added documentation, cleaned up code some
    Refactored FlowFileQueue so that there is SwappablePriorityQueue
    Several unit tests written
    Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles.
    Bug fixes
    Code refactoring
    Added integration tests, bug fixes
    Refactored clients to use NIO
    Bug fixes. Appears to finally be working with NIO Client!!!!!

commit 227fea81e6af1bd6d7d97ccb062c03c8c2377dc0
Author: Mark Payne <ma...@...>
Date:   2018-07-27T16:40:14Z

    NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession
    Bug fixes and allowed load balancing socket connections to be reused
    Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing
    Added flag to ConnectionDTO to indicate Load Balance Status
    Updated Diagnostics DTO for connections
    Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart
    Code cleanup
    Fixed checkstyle and unit tests

----


> Allow data in a Connection to be Load-Balanced across cluster
> -------------------------------------------------------------
>
>                 Key: NIFI-5516
>                 URL: https://issues.apache.org/jira/browse/NIFI-5516
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>
> Allow user to configure a Connection to be load balanced across the cluster. For more information, see Feature Proposal at https://cwiki.apache.org/confluence/display/NIFI/Load-Balanced+Connections



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)