[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224453#comment-15224453 ] 

ASF subversion and git services commented on NIFI-1563:
-------------------------------------------------------

Commit 0d3bd2c401aacc5bcd602afe90883d751ff773f3 in nifi's branch refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=0d3bd2c ]

NIFI-1563: - Federate requests and merge responses from nodes instead of storing bulletins and stats at NCM - Updating UI to support restructured status history DTO. - Return 'Insufficient History' message if aggregate stats don't have enough data points, even if all nodes do (which can be the case if the node performing the aggregation has a different value for the 'nifi.components.status.snapshot.frequency' property than the other nodes) - Bug fixes; code cleanup; replicate requests to bulletin board endpoint - Refactored the <Component>StatusDTO objects into <Component>StatusDTO, <Component>StatusSnapshotDTO, Node<Component>StatusSnapshotDTO objects - Introducing endpoints for accessing individual component status. - Wiring up new endpoints and updated core. - Code clean up. - Starting to handling status merging of individual components. - Nodewise breakdown has been added to Processors but the remaining components still need to be updated. - Refactor so that System Diagnostics requests are replicated to nodes instead of the information being pulled from Heartbeats - Replicate request for counters instead of pulling them from heartbeats - Removed the getCounters / setCounters method from HeartbeatPayload - Implementing component specific endpoints. - Removing unused endpoints. - Supporting nodewise breakdown for system diagnostics and counters. - Updating DTOs to use more consistent naming. - Code clean up. - Addressing contrib issues. - Removed ProcessGroupStatus from HeartbeatPayload - Removing nodewise from the system diagnostics endpoint. Had included it for testing that option but did not intend for it to be committed. - Addressing comments in PR #294. - This closes #294

Signed-off-by: Matt Gilman <ma...@gmail.com>


> Remove storage of components' stats and bulletins from NCM
> ----------------------------------------------------------
>
>                 Key: NIFI-1563
>                 URL: https://issues.apache.org/jira/browse/NIFI-1563
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 1.0.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Currently, each node in a cluster sends a period heartbeat that contains the stats for all components in the node. This happens every 5 seconds by default. This results in quite a lot of chatter between the NCM and nodes. It made sense to take this approach when the clustering concept was designed because there were no Process Groups, and we had no notion of merging responses from nodes for a web request.
> However, now we should replicate the request to all nodes and then merge the responses on demand, rather than storing this information on the NCM. This requires far less bandwidth because we only need to pull the stats for a particular Process Group and only on demand instead of every 5 seconds. 
> Additionally, this is laying groundwork for the Zero-Master clustering that we want to have in place for 1.0.0.
> In order to remove the stats from the NCM, we will also need to remove the bulletins and stats history. These requests will all need to be federated and the responses merged on-demand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)