[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM

["Mark Payne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167253#comment-15167253 ] 

Mark Payne commented on NIFI-1563:
----------------------------------

Will also require requests for Counters & System Diagnostics to be federated.

> Remove storage of components' stats and bulletins from NCM
> ----------------------------------------------------------
>
>                 Key: NIFI-1563
>                 URL: https://issues.apache.org/jira/browse/NIFI-1563
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>             Fix For: 1.0.0
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Currently, each node in a cluster sends a period heartbeat that contains the stats for all components in the node. This happens every 5 seconds by default. This results in quite a lot of chatter between the NCM and nodes. It made sense to take this approach when the clustering concept was designed because there were no Process Groups, and we had no notion of merging responses from nodes for a web request.
> However, now we should replicate the request to all nodes and then merge the responses on demand, rather than storing this information on the NCM. This requires far less bandwidth because we only need to pull the stats for a particular Process Group and only on demand instead of every 5 seconds. 
> Additionally, this is laying groundwork for the Zero-Master clustering that we want to have in place for 1.0.0.
> In order to remove the stats from the NCM, we will also need to remove the bulletins and stats history. These requests will all need to be federated and the responses merged on-demand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)