[jira] [Commented] (NIFI-338) The NCM should offer high availability

["Mark Payne (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670878#comment-15670878 ] 

Mark Payne commented on NIFI-338:
---------------------------------

[~nijel] - sorry I never saw your comment on this ticket. Looks like you commented quite a while back in April. At this point we have released 1.0, which provides a Zero-Master Clustering paradigm. The NCM is gone, and is now replaced by allowing Leader Election via ZooKeeper to handle the logic that previously required a single NCM.

[~apurtell] - I would like to close this ticket now, with 1.0.0 having been released. Are you okay with me closing it (or you could close it if you prefer)?

Thanks
-Mark

> The NCM should offer high availability
> --------------------------------------
>
>                 Key: NIFI-338
>                 URL: https://issues.apache.org/jira/browse/NIFI-338
>             Project: Apache NiFi
>          Issue Type: Sub-task
>          Components: Core Framework
>            Reporter: Andrew Purtell
>
> Purtell:
>  ​I'm separately curious about the level of effort it
> would require to introduce multiple masters.​ I think of like how HBase
> does it: there is only ever one active master but a user can deploy
> multiple standbys to take over service should the active master fail.
> Witt: 
> So it is true that we have a single master model today.  But that
> single master is solely for command/control of changes to the dataflow
> configuration and is a very lightweight process that does nothing more
> than that.  If the master dies then all nodes continue to do what they
> were doing and even site-to-site continues to distribute data.  It
> just does so without updates on current loading across the cluster.
> Once the master is brought back on-line then the real-time command and
> control functions return.  Building support for a back-up master to
> offer HA of even the command/control side would probably also be a
> considerable effort.  This one I'd be curious to hear of cases where
> it was critical to make this part HA.
> Purtell:
> Yes but imagine a NiFi installation, perhaps a hosted service built on top
> of it, where DataFlow Managers expect the command and control aspect of the
> system to be as robust and available as flow processing itself. If one or
> more standby masters are waiting in the wings to take over service for the
> failed active master then automated and unattended failover would be
> possible, and likely to narrow the interval where administrative changes
> may fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)