[DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Matt Burgess <ma...@apache.org>]

Hello all,

I'd like to start a discussion about the current and future Command
and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
codebase. The current implementation[1] is IMO not a pure reference
implementation of the C2 spec [2]. It has fairly limited capability
and uses the PullHttpChangeIngestor for the agent to ask for flow
updates, which makes any bidirectional communication difficult. Rather
than try to build out from what we currently have, I propose a new
implementation (deprecating and eventually removing the old one) that
is more generic, extensible, etc. This would include the following
options:

- Renaming of the minifi-c2 modules and perhaps the release artifacts,
updating the documentation accordingly
- Creation of a top-level "c2" or "nifi-c2" module that includes an
API module, assembly, NAR, implementation, services bundle, etc. as
needed
- Adding/deleting/changing/using properties related to C2 configuration
- Changing the common runtime/loading code to look for a C2 client
implementation and instantiate that at startup, configuring it with
the appropriate properties and injecting the objects needed for C2
capabilities
- Implementing a new C2 server that supports more of the C2 spec, and
making adding more of the commands easier going forward

I propose the Minimum Viable Product (MVP) C2 Client would have the
following capabilities:
- A "Manifest Reporter" that can either push a manifest (a document
containing all the agent/instance capabilities such as available
processors and their metadata, e.g.) and/or respond to a DESCRIBE
MANIFEST request (see [2])
- A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply
with ACKNOWLEDGEMENT messages and dispatching any commands such as
UPDATE
- The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in
the old system, but is much more capable
- Supports HTTPS

I propose the MVP C2 Server would have the following capabilities:
- Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST
- Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially
using the same approach as the old C2 system (loading the flow from a
file for the UPDATE command) but would be extensible/configurable
- Supports HTTPS

This should give us not only feature parity with the old C2 system,
but also presents a framework to build out the rest of the C2 design
and add other capabilities. I should note that this proposal is not
specific to MiNiFi Java. It should aim to align and use the same
protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2
capabilities to NiFi itself in the future. We could track the proposal
and progress at [4] as I believe that's what the page is intended for.

I'd like to hear your thoughts on these proposals, and I'm happy to
answer any questions or clarify as necessary. Also as this proceeds
I'm happy to write up any Jira epics/cases/etc. and do the initial
project structure and API setup work so others can dig in.

Regards,
Matt

[1] https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md
[2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design
[3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md
[4] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108

Re: [DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Marton Szasz <sz...@apache.org>]

Hi,

I like the idea of using the same protocol to control NiFi, MiNiFi
Java and MiNiFi C++.

I see no issues with supporting VersionedFlowSnapshot as a config
format for MiNiFi C++ as well, but I only checked the schema so far.
The C++ agent needs to get a new config parser written from scratch to
support this format, but it's doable IMO. Since there are some minor
functional differences between the Java and the C++ processors, I'm
not confident in minifi-toolkit's ability to convert complex flows to
the config.yml format of MiNiFi C++, and for this reason, I want to
keep the YAML config support for manual flow authoring with a text
editor.

Process control and the RESTART command is hard to implement in a
robust way. In MiNiFi C++ currently it's implemented as a hardcoded
shell call to `bin/minifi.sh restart`, but the windows service,
minifi.bat, systemd service, docker container and other cases are not
handled. Maybe we should let the administrator specify a restart
script in the properties file for a more robust handling of the
command. That's the least janky command implementation that comes to
my mind. We could also fork on startup and let the parent restart the
child, but I prefer not to keep a sleeping background process just for
this purpose. Since RESTART is already part of the protocol, at least
on the C++ side, so I see no reason not to keep it and implement
something similar in MiNiFi Java, even if it's less than perfect.

Thanks,
Marton




On Tue, 30 Nov 2021 at 18:56, Kevin Doran <kd...@apache.org> wrote:
>
> Hi Matt,
>
> Thanks for kicking of this thread. The discussion here puts into words some things that have been on my mind for a while, and its really exciting to see a revival of these conversations.
>
> Overall, I support the proposal. Here are some specific thoughts, in no particular order.
>
> One of the valuable things I see emerging in the NiFi ecosystem is the separation of flow design and flow runtime, with runtime options being NiFi, MiNiFi (C++/Java), and NiFi Stateless — and that is before you even look at the underlying infrastructure options such as bare metal, virtual, containerized, or server-less —  so standardizing a portable, serialized format of flow definitions seems like a huge advantage for the ecosystem. The VersionedFlowSnapshot has emerged as the best option for that. I see there is work on NiFi to even replace the flow.xml.gz with this format, so updating MiNiFi to support it as well as an alternative (and perhaps eventually replacement) for config.yml would be in line with that.
>
> I think a reference Java implementation of the C2 Spec implemented by the MiNiFi C++ agent would be a huge step forward for the goal of unified C2 across the NiFi ecosystem. For the capabilities we would like to support, the MiNiFi C++ C2 Protocol is the correct one to standardize on in order to support the wide array of C2 capabilities we would like to (compared to, for example, the PullHttpChangeIngestor currently supported by MiNiFi Java).
>
> Regarding process control vs. flow / metadata control: I think the C2 Protocol Spec is flexible enough to support both, and that how a C2 client implements support for C2 operation commands is an implementation detail for the client. I know that the current MiNiFi C++ client can handle both process and flow control, and I don’t think that needs to change. I think for the MVP for the proposed MiNiFi Java client, focusing on flow control makes sense. I agree with Marc that beyond the MVP, process control of a MiNiFi Java agent may be desirable, but I don’t think the proposed approach makes that infeasible. Because the Bootstrap<->MiNiFi control bus is bidirectional, I could foresee the C2 client in the MiNiFi process forwarding process control operations to a C2 helper in the bootstrap process. In other words, a C2 client in the core (Mi)NiFi framework could send a message to bootstrap saying “please change my config to X and restart me”. I view the C2 Protocol as being a superset of all possible functionality a NiFi Runtime could support, and process control commands might not even exist for a runtime such as NiFi Stateless on serverless infrastructure (but metadata exchange might), so such a runtime that wants to implement a subset of the C2 Protocol operations could just choose not to support such things that don’t make sense.
>
> One last thought: anything that more closely aligns the implementation of MiNiFi Java and NiFi is a big step in the right direction, so building this C2 capability out in NiFi such that we can take advantage of it in MiNiFi Java seems to be the best thing to do. If NiFi is able to run with a smaller footprint and pull in extensions at runtime, then supporting C2 in NiFi proper probably also makes sense as a way of central control for say, a fleet of containerized NiFi instances.
>
> Thanks,
> Kevin
>
>
>
> > On Nov 22, 2021, at 10:03, Matt Burgess <ma...@apache.org> wrote:
> >
> > I'm the same, historically I've viewed and implemented C2 as process
> > control as well. These days with sophisticated containerization and
> > deployment tools, I'm thinking we leave the process orchestration to
> > them and concentrate on controlling what the processes are doing.
> >
> > All, any other thoughts, questions, or concerns? If not, I will update
> > the C2 page(s) on the Wiki as discussed and start writing up some
> > Jiras to cover the work, and we can continue individual feature
> > discussions there.
> >
> > Thanks,
> > Matt
> >
> > On Sat, Nov 20, 2021 at 6:19 AM Marc Parisi <ph...@apache.org> wrote:
> >>
> >> " This would align it more to the way NiFi works as
> >> well, allowing C2 to be used for flow control and metadata exchange
> >> rather than process control."
> >>
> >> This particularly caught my eye and is something that makes sense from an
> >> evolutionary perspective.
> >>
> >> I've previously viewed command and control as a mechanism to control the
> >> agent processes and their flows.
> >>
> >> From an operational perspective, I think that agent process control would
> >> be nice to have (if not a necessity for some ); however, if C2 is scoped to
> >> metadata and flow control at least -- through the MVP or permanently --
> >> then the conversation is simpler and I think your proposal is great.
> >>
> >> Thanks for the additional insight, Matt.
> >>
> >> Thanks,
> >> Marc
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Nov 18, 2021 at 4:37 PM Matt Burgess <ma...@apache.org> wrote:
> >>
> >>> Marc,
> >>>
> >>> That's a great point and reminds me that I missed two whole sections
> >>> of the proposal: 1) The refactor of minifi-bootstrap, and 2) the
> >>> standardization of serialized flow control to VersionedFlowSnapshot
> >>> (i.e. JSON not XML or YAML).
> >>>
> >>> 1) Currently it's the bootstrap process that reaches out to the C2
> >>> server and handles the commands such as UPDATE and RESTART. In fact,
> >>> UPDATE updates the flow on-disk and RESTART then kills and restarts
> >>> the MiNiFi process.  I think with the time that's passed since its
> >>> inception, it has kind of gotten bloated and has taken on capabilities
> >>> probably best left to a separate module, since those things make it
> >>> inconsistent with the way NiFi bootstrap works . In that vein I'd
> >>> propose (at some point, not necessarily MVP but that'd be great) that
> >>> we standardize the bootstraps as much as possible, and put all the C2
> >>> stuff into the C2 modules (i.e. the C2 Client in the case of
> >>> minifi-bootstrap). I vaguely alluded to that when I said the C2 client
> >>> would need access to the internal objects, but that glossed WAY over
> >>> the actual implication, which is that the C2 client module will be the
> >>> thing talking to the C2 server to handle all the C2 messages including
> >>> UPDATE, and I think RESTART might be replaced with RELOAD or absorbed
> >>> into UPDATE, but happy to discuss that as well.
> >>>
> >>> If there are issues such as updating an agent/instance with
> >>> configuration properties, I anticipate a new proposal and design to
> >>> cover that, but I invite current C2 Server users to jump in here
> >>> please. As it stands I believe we haven't implemented enough of the C2
> >>> capabilities to be affected, unless the C2 server's flow update file
> >>> includes configurable property updates. Otherwise I would think just a
> >>> flow update would suffice, and could be done without restarting the
> >>> process itself. This would align it more to the way NiFi works as
> >>> well, allowing C2 to be used for flow control and metadata exchange
> >>> rather than process control.
> >>>
> >>> 2) IIRC, the "config.yml' approach came from two use cases, to be able
> >>> to update the agent process/configuration itself and to make it easier
> >>> to "design" a flow by having more user-friendly names for the various
> >>> configuration and component properties without a UI designer. These
> >>> days, the minifi-toolkit allows the user to convert a flow from
> >>> flow.xml.gz to a config.yml, and I'm aware of at least one vendor
> >>> product for flow design & publish via a GUI. Between the
> >>> aforementioned lack of a need to update the agent process, and ongoing
> >>> work [1] to standardize the flow definition for NiFi, Registry,
> >>> Stateless, etc. to VersionedFlowSnapshot, I think MiNiFi should follow
> >>> suit. I can't speak to the C++ side of things but perhaps those more
> >>> familiar with that project can weigh in.
> >>>
> >>> My original email turned out to be more focused on the MVP products,
> >>> but these additional aspects (as well as any others the community
> >>> would like to offer) should be discussed too. If these are not
> >>> necessary for MVP, perhaps we can get the work going on the API and
> >>> MVP and continue the discussions on the Wiki and/or mailing list as
> >>> folks want to contribute the extra goodies :)
> >>>
> >>> Thanks,
> >>> Matt
> >>>
> >>> [1] https://issues.apache.org/jira/browse/NIFI-9069
> >>>
> >>> On Thu, Nov 18, 2021 at 4:07 PM Marc Parisi <ph...@apache.org> wrote:
> >>>>
> >>>> Hi Matt,
> >>>>
> >>>> I think this is great. I like the MVP set of features.
> >>>>
> >>>> Considering use cases and potential maintenance needs, are there thoughts
> >>>> on including RESTART in the MVP for cases where agent UPDATES may be
> >>>> blocked for whatever reason ( I/O, agent problems, etc )?
> >>>>
> >>>> Does the risk of the RESTART verb being somewhat nebulous outweigh any
> >>>> benefits?
> >>>>
> >>>> Again, great stuff!
> >>>>
> >>>> Thanks,
> >>>> Marc
> >>>>
> >>>>
> >>>> On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess <ma...@apache.org>
> >>> wrote:
> >>>>
> >>>>> Hello all,
> >>>>>
> >>>>> I'd like to start a discussion about the current and future Command
> >>>>> and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
> >>>>> codebase. The current implementation[1] is IMO not a pure reference
> >>>>> implementation of the C2 spec [2]. It has fairly limited capability
> >>>>> and uses the PullHttpChangeIngestor for the agent to ask for flow
> >>>>> updates, which makes any bidirectional communication difficult. Rather
> >>>>> than try to build out from what we currently have, I propose a new
> >>>>> implementation (deprecating and eventually removing the old one) that
> >>>>> is more generic, extensible, etc. This would include the following
> >>>>> options:
> >>>>>
> >>>>> - Renaming of the minifi-c2 modules and perhaps the release artifacts,
> >>>>> updating the documentation accordingly
> >>>>> - Creation of a top-level "c2" or "nifi-c2" module that includes an
> >>>>> API module, assembly, NAR, implementation, services bundle, etc. as
> >>>>> needed
> >>>>> - Adding/deleting/changing/using properties related to C2 configuration
> >>>>> - Changing the common runtime/loading code to look for a C2 client
> >>>>> implementation and instantiate that at startup, configuring it with
> >>>>> the appropriate properties and injecting the objects needed for C2
> >>>>> capabilities
> >>>>> - Implementing a new C2 server that supports more of the C2 spec, and
> >>>>> making adding more of the commands easier going forward
> >>>>>
> >>>>> I propose the Minimum Viable Product (MVP) C2 Client would have the
> >>>>> following capabilities:
> >>>>> - A "Manifest Reporter" that can either push a manifest (a document
> >>>>> containing all the agent/instance capabilities such as available
> >>>>> processors and their metadata, e.g.) and/or respond to a DESCRIBE
> >>>>> MANIFEST request (see [2])
> >>>>> - A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply
> >>>>> with ACKNOWLEDGEMENT messages and dispatching any commands such as
> >>>>> UPDATE
> >>>>> - The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in
> >>>>> the old system, but is much more capable
> >>>>> - Supports HTTPS
> >>>>>
> >>>>> I propose the MVP C2 Server would have the following capabilities:
> >>>>> - Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST
> >>>>> - Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially
> >>>>> using the same approach as the old C2 system (loading the flow from a
> >>>>> file for the UPDATE command) but would be extensible/configurable
> >>>>> - Supports HTTPS
> >>>>>
> >>>>> This should give us not only feature parity with the old C2 system,
> >>>>> but also presents a framework to build out the rest of the C2 design
> >>>>> and add other capabilities. I should note that this proposal is not
> >>>>> specific to MiNiFi Java. It should aim to align and use the same
> >>>>> protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2
> >>>>> capabilities to NiFi itself in the future. We could track the proposal
> >>>>> and progress at [4] as I believe that's what the page is intended for.
> >>>>>
> >>>>> I'd like to hear your thoughts on these proposals, and I'm happy to
> >>>>> answer any questions or clarify as necessary. Also as this proceeds
> >>>>> I'm happy to write up any Jira epics/cases/etc. and do the initial
> >>>>> project structure and API setup work so others can dig in.
> >>>>>
> >>>>> Regards,
> >>>>> Matt
> >>>>>
> >>>>> [1]
> >>> https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md
> >>>>> [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design
> >>>>> [3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md
> >>>>> [4]
> >>>>>
> >>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108
> >>>>>
> >>>
>

Re: [DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Kevin Doran <kd...@apache.org>]

Hi Matt,

Thanks for kicking of this thread. The discussion here puts into words some things that have been on my mind for a while, and its really exciting to see a revival of these conversations. 

Overall, I support the proposal. Here are some specific thoughts, in no particular order.

One of the valuable things I see emerging in the NiFi ecosystem is the separation of flow design and flow runtime, with runtime options being NiFi, MiNiFi (C++/Java), and NiFi Stateless — and that is before you even look at the underlying infrastructure options such as bare metal, virtual, containerized, or server-less —  so standardizing a portable, serialized format of flow definitions seems like a huge advantage for the ecosystem. The VersionedFlowSnapshot has emerged as the best option for that. I see there is work on NiFi to even replace the flow.xml.gz with this format, so updating MiNiFi to support it as well as an alternative (and perhaps eventually replacement) for config.yml would be in line with that.

I think a reference Java implementation of the C2 Spec implemented by the MiNiFi C++ agent would be a huge step forward for the goal of unified C2 across the NiFi ecosystem. For the capabilities we would like to support, the MiNiFi C++ C2 Protocol is the correct one to standardize on in order to support the wide array of C2 capabilities we would like to (compared to, for example, the PullHttpChangeIngestor currently supported by MiNiFi Java).

Regarding process control vs. flow / metadata control: I think the C2 Protocol Spec is flexible enough to support both, and that how a C2 client implements support for C2 operation commands is an implementation detail for the client. I know that the current MiNiFi C++ client can handle both process and flow control, and I don’t think that needs to change. I think for the MVP for the proposed MiNiFi Java client, focusing on flow control makes sense. I agree with Marc that beyond the MVP, process control of a MiNiFi Java agent may be desirable, but I don’t think the proposed approach makes that infeasible. Because the Bootstrap<->MiNiFi control bus is bidirectional, I could foresee the C2 client in the MiNiFi process forwarding process control operations to a C2 helper in the bootstrap process. In other words, a C2 client in the core (Mi)NiFi framework could send a message to bootstrap saying “please change my config to X and restart me”. I view the C2 Protocol as being a superset of all possible functionality a NiFi Runtime could support, and process control commands might not even exist for a runtime such as NiFi Stateless on serverless infrastructure (but metadata exchange might), so such a runtime that wants to implement a subset of the C2 Protocol operations could just choose not to support such things that don’t make sense.

One last thought: anything that more closely aligns the implementation of MiNiFi Java and NiFi is a big step in the right direction, so building this C2 capability out in NiFi such that we can take advantage of it in MiNiFi Java seems to be the best thing to do. If NiFi is able to run with a smaller footprint and pull in extensions at runtime, then supporting C2 in NiFi proper probably also makes sense as a way of central control for say, a fleet of containerized NiFi instances.

Thanks,
Kevin



> On Nov 22, 2021, at 10:03, Matt Burgess <ma...@apache.org> wrote:
> 
> I'm the same, historically I've viewed and implemented C2 as process
> control as well. These days with sophisticated containerization and
> deployment tools, I'm thinking we leave the process orchestration to
> them and concentrate on controlling what the processes are doing.
> 
> All, any other thoughts, questions, or concerns? If not, I will update
> the C2 page(s) on the Wiki as discussed and start writing up some
> Jiras to cover the work, and we can continue individual feature
> discussions there.
> 
> Thanks,
> Matt
> 
> On Sat, Nov 20, 2021 at 6:19 AM Marc Parisi <ph...@apache.org> wrote:
>> 
>> " This would align it more to the way NiFi works as
>> well, allowing C2 to be used for flow control and metadata exchange
>> rather than process control."
>> 
>> This particularly caught my eye and is something that makes sense from an
>> evolutionary perspective.
>> 
>> I've previously viewed command and control as a mechanism to control the
>> agent processes and their flows.
>> 
>> From an operational perspective, I think that agent process control would
>> be nice to have (if not a necessity for some ); however, if C2 is scoped to
>> metadata and flow control at least -- through the MVP or permanently --
>> then the conversation is simpler and I think your proposal is great.
>> 
>> Thanks for the additional insight, Matt.
>> 
>> Thanks,
>> Marc
>> 
>> 
>> 
>> 
>> 
>> On Thu, Nov 18, 2021 at 4:37 PM Matt Burgess <ma...@apache.org> wrote:
>> 
>>> Marc,
>>> 
>>> That's a great point and reminds me that I missed two whole sections
>>> of the proposal: 1) The refactor of minifi-bootstrap, and 2) the
>>> standardization of serialized flow control to VersionedFlowSnapshot
>>> (i.e. JSON not XML or YAML).
>>> 
>>> 1) Currently it's the bootstrap process that reaches out to the C2
>>> server and handles the commands such as UPDATE and RESTART. In fact,
>>> UPDATE updates the flow on-disk and RESTART then kills and restarts
>>> the MiNiFi process.  I think with the time that's passed since its
>>> inception, it has kind of gotten bloated and has taken on capabilities
>>> probably best left to a separate module, since those things make it
>>> inconsistent with the way NiFi bootstrap works . In that vein I'd
>>> propose (at some point, not necessarily MVP but that'd be great) that
>>> we standardize the bootstraps as much as possible, and put all the C2
>>> stuff into the C2 modules (i.e. the C2 Client in the case of
>>> minifi-bootstrap). I vaguely alluded to that when I said the C2 client
>>> would need access to the internal objects, but that glossed WAY over
>>> the actual implication, which is that the C2 client module will be the
>>> thing talking to the C2 server to handle all the C2 messages including
>>> UPDATE, and I think RESTART might be replaced with RELOAD or absorbed
>>> into UPDATE, but happy to discuss that as well.
>>> 
>>> If there are issues such as updating an agent/instance with
>>> configuration properties, I anticipate a new proposal and design to
>>> cover that, but I invite current C2 Server users to jump in here
>>> please. As it stands I believe we haven't implemented enough of the C2
>>> capabilities to be affected, unless the C2 server's flow update file
>>> includes configurable property updates. Otherwise I would think just a
>>> flow update would suffice, and could be done without restarting the
>>> process itself. This would align it more to the way NiFi works as
>>> well, allowing C2 to be used for flow control and metadata exchange
>>> rather than process control.
>>> 
>>> 2) IIRC, the "config.yml' approach came from two use cases, to be able
>>> to update the agent process/configuration itself and to make it easier
>>> to "design" a flow by having more user-friendly names for the various
>>> configuration and component properties without a UI designer. These
>>> days, the minifi-toolkit allows the user to convert a flow from
>>> flow.xml.gz to a config.yml, and I'm aware of at least one vendor
>>> product for flow design & publish via a GUI. Between the
>>> aforementioned lack of a need to update the agent process, and ongoing
>>> work [1] to standardize the flow definition for NiFi, Registry,
>>> Stateless, etc. to VersionedFlowSnapshot, I think MiNiFi should follow
>>> suit. I can't speak to the C++ side of things but perhaps those more
>>> familiar with that project can weigh in.
>>> 
>>> My original email turned out to be more focused on the MVP products,
>>> but these additional aspects (as well as any others the community
>>> would like to offer) should be discussed too. If these are not
>>> necessary for MVP, perhaps we can get the work going on the API and
>>> MVP and continue the discussions on the Wiki and/or mailing list as
>>> folks want to contribute the extra goodies :)
>>> 
>>> Thanks,
>>> Matt
>>> 
>>> [1] https://issues.apache.org/jira/browse/NIFI-9069
>>> 
>>> On Thu, Nov 18, 2021 at 4:07 PM Marc Parisi <ph...@apache.org> wrote:
>>>> 
>>>> Hi Matt,
>>>> 
>>>> I think this is great. I like the MVP set of features.
>>>> 
>>>> Considering use cases and potential maintenance needs, are there thoughts
>>>> on including RESTART in the MVP for cases where agent UPDATES may be
>>>> blocked for whatever reason ( I/O, agent problems, etc )?
>>>> 
>>>> Does the risk of the RESTART verb being somewhat nebulous outweigh any
>>>> benefits?
>>>> 
>>>> Again, great stuff!
>>>> 
>>>> Thanks,
>>>> Marc
>>>> 
>>>> 
>>>> On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess <ma...@apache.org>
>>> wrote:
>>>> 
>>>>> Hello all,
>>>>> 
>>>>> I'd like to start a discussion about the current and future Command
>>>>> and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
>>>>> codebase. The current implementation[1] is IMO not a pure reference
>>>>> implementation of the C2 spec [2]. It has fairly limited capability
>>>>> and uses the PullHttpChangeIngestor for the agent to ask for flow
>>>>> updates, which makes any bidirectional communication difficult. Rather
>>>>> than try to build out from what we currently have, I propose a new
>>>>> implementation (deprecating and eventually removing the old one) that
>>>>> is more generic, extensible, etc. This would include the following
>>>>> options:
>>>>> 
>>>>> - Renaming of the minifi-c2 modules and perhaps the release artifacts,
>>>>> updating the documentation accordingly
>>>>> - Creation of a top-level "c2" or "nifi-c2" module that includes an
>>>>> API module, assembly, NAR, implementation, services bundle, etc. as
>>>>> needed
>>>>> - Adding/deleting/changing/using properties related to C2 configuration
>>>>> - Changing the common runtime/loading code to look for a C2 client
>>>>> implementation and instantiate that at startup, configuring it with
>>>>> the appropriate properties and injecting the objects needed for C2
>>>>> capabilities
>>>>> - Implementing a new C2 server that supports more of the C2 spec, and
>>>>> making adding more of the commands easier going forward
>>>>> 
>>>>> I propose the Minimum Viable Product (MVP) C2 Client would have the
>>>>> following capabilities:
>>>>> - A "Manifest Reporter" that can either push a manifest (a document
>>>>> containing all the agent/instance capabilities such as available
>>>>> processors and their metadata, e.g.) and/or respond to a DESCRIBE
>>>>> MANIFEST request (see [2])
>>>>> - A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply
>>>>> with ACKNOWLEDGEMENT messages and dispatching any commands such as
>>>>> UPDATE
>>>>> - The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in
>>>>> the old system, but is much more capable
>>>>> - Supports HTTPS
>>>>> 
>>>>> I propose the MVP C2 Server would have the following capabilities:
>>>>> - Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST
>>>>> - Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially
>>>>> using the same approach as the old C2 system (loading the flow from a
>>>>> file for the UPDATE command) but would be extensible/configurable
>>>>> - Supports HTTPS
>>>>> 
>>>>> This should give us not only feature parity with the old C2 system,
>>>>> but also presents a framework to build out the rest of the C2 design
>>>>> and add other capabilities. I should note that this proposal is not
>>>>> specific to MiNiFi Java. It should aim to align and use the same
>>>>> protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2
>>>>> capabilities to NiFi itself in the future. We could track the proposal
>>>>> and progress at [4] as I believe that's what the page is intended for.
>>>>> 
>>>>> I'd like to hear your thoughts on these proposals, and I'm happy to
>>>>> answer any questions or clarify as necessary. Also as this proceeds
>>>>> I'm happy to write up any Jira epics/cases/etc. and do the initial
>>>>> project structure and API setup work so others can dig in.
>>>>> 
>>>>> Regards,
>>>>> Matt
>>>>> 
>>>>> [1]
>>> https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md
>>>>> [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design
>>>>> [3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md
>>>>> [4]
>>>>> 
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108
>>>>> 
>>> 

Re: [DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Matt Burgess <ma...@apache.org>]

I'm the same, historically I've viewed and implemented C2 as process
control as well. These days with sophisticated containerization and
deployment tools, I'm thinking we leave the process orchestration to
them and concentrate on controlling what the processes are doing.

All, any other thoughts, questions, or concerns? If not, I will update
the C2 page(s) on the Wiki as discussed and start writing up some
Jiras to cover the work, and we can continue individual feature
discussions there.

Thanks,
Matt

On Sat, Nov 20, 2021 at 6:19 AM Marc Parisi <ph...@apache.org> wrote:
>
> " This would align it more to the way NiFi works as
> well, allowing C2 to be used for flow control and metadata exchange
> rather than process control."
>
> This particularly caught my eye and is something that makes sense from an
> evolutionary perspective.
>
> I've previously viewed command and control as a mechanism to control the
> agent processes and their flows.
>
> From an operational perspective, I think that agent process control would
> be nice to have (if not a necessity for some ); however, if C2 is scoped to
> metadata and flow control at least -- through the MVP or permanently --
> then the conversation is simpler and I think your proposal is great.
>
> Thanks for the additional insight, Matt.
>
> Thanks,
> Marc
>
>
>
>
>
> On Thu, Nov 18, 2021 at 4:37 PM Matt Burgess <ma...@apache.org> wrote:
>
> > Marc,
> >
> > That's a great point and reminds me that I missed two whole sections
> > of the proposal: 1) The refactor of minifi-bootstrap, and 2) the
> > standardization of serialized flow control to VersionedFlowSnapshot
> > (i.e. JSON not XML or YAML).
> >
> > 1) Currently it's the bootstrap process that reaches out to the C2
> > server and handles the commands such as UPDATE and RESTART. In fact,
> > UPDATE updates the flow on-disk and RESTART then kills and restarts
> > the MiNiFi process.  I think with the time that's passed since its
> > inception, it has kind of gotten bloated and has taken on capabilities
> > probably best left to a separate module, since those things make it
> > inconsistent with the way NiFi bootstrap works . In that vein I'd
> > propose (at some point, not necessarily MVP but that'd be great) that
> > we standardize the bootstraps as much as possible, and put all the C2
> > stuff into the C2 modules (i.e. the C2 Client in the case of
> > minifi-bootstrap). I vaguely alluded to that when I said the C2 client
> > would need access to the internal objects, but that glossed WAY over
> > the actual implication, which is that the C2 client module will be the
> > thing talking to the C2 server to handle all the C2 messages including
> > UPDATE, and I think RESTART might be replaced with RELOAD or absorbed
> > into UPDATE, but happy to discuss that as well.
> >
> > If there are issues such as updating an agent/instance with
> > configuration properties, I anticipate a new proposal and design to
> > cover that, but I invite current C2 Server users to jump in here
> > please. As it stands I believe we haven't implemented enough of the C2
> > capabilities to be affected, unless the C2 server's flow update file
> > includes configurable property updates. Otherwise I would think just a
> > flow update would suffice, and could be done without restarting the
> > process itself. This would align it more to the way NiFi works as
> > well, allowing C2 to be used for flow control and metadata exchange
> > rather than process control.
> >
> > 2) IIRC, the "config.yml' approach came from two use cases, to be able
> > to update the agent process/configuration itself and to make it easier
> > to "design" a flow by having more user-friendly names for the various
> > configuration and component properties without a UI designer. These
> > days, the minifi-toolkit allows the user to convert a flow from
> > flow.xml.gz to a config.yml, and I'm aware of at least one vendor
> > product for flow design & publish via a GUI. Between the
> > aforementioned lack of a need to update the agent process, and ongoing
> > work [1] to standardize the flow definition for NiFi, Registry,
> > Stateless, etc. to VersionedFlowSnapshot, I think MiNiFi should follow
> > suit. I can't speak to the C++ side of things but perhaps those more
> > familiar with that project can weigh in.
> >
> > My original email turned out to be more focused on the MVP products,
> > but these additional aspects (as well as any others the community
> > would like to offer) should be discussed too. If these are not
> > necessary for MVP, perhaps we can get the work going on the API and
> > MVP and continue the discussions on the Wiki and/or mailing list as
> > folks want to contribute the extra goodies :)
> >
> > Thanks,
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-9069
> >
> > On Thu, Nov 18, 2021 at 4:07 PM Marc Parisi <ph...@apache.org> wrote:
> > >
> > > Hi Matt,
> > >
> > > I think this is great. I like the MVP set of features.
> > >
> > > Considering use cases and potential maintenance needs, are there thoughts
> > > on including RESTART in the MVP for cases where agent UPDATES may be
> > > blocked for whatever reason ( I/O, agent problems, etc )?
> > >
> > > Does the risk of the RESTART verb being somewhat nebulous outweigh any
> > > benefits?
> > >
> > > Again, great stuff!
> > >
> > > Thanks,
> > > Marc
> > >
> > >
> > > On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess <ma...@apache.org>
> > wrote:
> > >
> > > > Hello all,
> > > >
> > > > I'd like to start a discussion about the current and future Command
> > > > and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
> > > > codebase. The current implementation[1] is IMO not a pure reference
> > > > implementation of the C2 spec [2]. It has fairly limited capability
> > > > and uses the PullHttpChangeIngestor for the agent to ask for flow
> > > > updates, which makes any bidirectional communication difficult. Rather
> > > > than try to build out from what we currently have, I propose a new
> > > > implementation (deprecating and eventually removing the old one) that
> > > > is more generic, extensible, etc. This would include the following
> > > > options:
> > > >
> > > > - Renaming of the minifi-c2 modules and perhaps the release artifacts,
> > > > updating the documentation accordingly
> > > > - Creation of a top-level "c2" or "nifi-c2" module that includes an
> > > > API module, assembly, NAR, implementation, services bundle, etc. as
> > > > needed
> > > > - Adding/deleting/changing/using properties related to C2 configuration
> > > > - Changing the common runtime/loading code to look for a C2 client
> > > > implementation and instantiate that at startup, configuring it with
> > > > the appropriate properties and injecting the objects needed for C2
> > > > capabilities
> > > > - Implementing a new C2 server that supports more of the C2 spec, and
> > > > making adding more of the commands easier going forward
> > > >
> > > > I propose the Minimum Viable Product (MVP) C2 Client would have the
> > > > following capabilities:
> > > > - A "Manifest Reporter" that can either push a manifest (a document
> > > > containing all the agent/instance capabilities such as available
> > > > processors and their metadata, e.g.) and/or respond to a DESCRIBE
> > > > MANIFEST request (see [2])
> > > > - A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply
> > > > with ACKNOWLEDGEMENT messages and dispatching any commands such as
> > > > UPDATE
> > > > - The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in
> > > > the old system, but is much more capable
> > > > - Supports HTTPS
> > > >
> > > > I propose the MVP C2 Server would have the following capabilities:
> > > > - Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST
> > > > - Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially
> > > > using the same approach as the old C2 system (loading the flow from a
> > > > file for the UPDATE command) but would be extensible/configurable
> > > > - Supports HTTPS
> > > >
> > > > This should give us not only feature parity with the old C2 system,
> > > > but also presents a framework to build out the rest of the C2 design
> > > > and add other capabilities. I should note that this proposal is not
> > > > specific to MiNiFi Java. It should aim to align and use the same
> > > > protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2
> > > > capabilities to NiFi itself in the future. We could track the proposal
> > > > and progress at [4] as I believe that's what the page is intended for.
> > > >
> > > > I'd like to hear your thoughts on these proposals, and I'm happy to
> > > > answer any questions or clarify as necessary. Also as this proceeds
> > > > I'm happy to write up any Jira epics/cases/etc. and do the initial
> > > > project structure and API setup work so others can dig in.
> > > >
> > > > Regards,
> > > > Matt
> > > >
> > > > [1]
> > https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md
> > > > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design
> > > > [3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md
> > > > [4]
> > > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108
> > > >
> >

Re: [DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Marc Parisi <ph...@apache.org>]

" This would align it more to the way NiFi works as
well, allowing C2 to be used for flow control and metadata exchange
rather than process control."

This particularly caught my eye and is something that makes sense from an
evolutionary perspective.

I've previously viewed command and control as a mechanism to control the
agent processes and their flows.

From an operational perspective, I think that agent process control would
be nice to have (if not a necessity for some ); however, if C2 is scoped to
metadata and flow control at least -- through the MVP or permanently --
then the conversation is simpler and I think your proposal is great.

Thanks for the additional insight, Matt.

Thanks,
Marc





On Thu, Nov 18, 2021 at 4:37 PM Matt Burgess <ma...@apache.org> wrote:

> Marc,
>
> That's a great point and reminds me that I missed two whole sections
> of the proposal: 1) The refactor of minifi-bootstrap, and 2) the
> standardization of serialized flow control to VersionedFlowSnapshot
> (i.e. JSON not XML or YAML).
>
> 1) Currently it's the bootstrap process that reaches out to the C2
> server and handles the commands such as UPDATE and RESTART. In fact,
> UPDATE updates the flow on-disk and RESTART then kills and restarts
> the MiNiFi process.  I think with the time that's passed since its
> inception, it has kind of gotten bloated and has taken on capabilities
> probably best left to a separate module, since those things make it
> inconsistent with the way NiFi bootstrap works . In that vein I'd
> propose (at some point, not necessarily MVP but that'd be great) that
> we standardize the bootstraps as much as possible, and put all the C2
> stuff into the C2 modules (i.e. the C2 Client in the case of
> minifi-bootstrap). I vaguely alluded to that when I said the C2 client
> would need access to the internal objects, but that glossed WAY over
> the actual implication, which is that the C2 client module will be the
> thing talking to the C2 server to handle all the C2 messages including
> UPDATE, and I think RESTART might be replaced with RELOAD or absorbed
> into UPDATE, but happy to discuss that as well.
>
> If there are issues such as updating an agent/instance with
> configuration properties, I anticipate a new proposal and design to
> cover that, but I invite current C2 Server users to jump in here
> please. As it stands I believe we haven't implemented enough of the C2
> capabilities to be affected, unless the C2 server's flow update file
> includes configurable property updates. Otherwise I would think just a
> flow update would suffice, and could be done without restarting the
> process itself. This would align it more to the way NiFi works as
> well, allowing C2 to be used for flow control and metadata exchange
> rather than process control.
>
> 2) IIRC, the "config.yml' approach came from two use cases, to be able
> to update the agent process/configuration itself and to make it easier
> to "design" a flow by having more user-friendly names for the various
> configuration and component properties without a UI designer. These
> days, the minifi-toolkit allows the user to convert a flow from
> flow.xml.gz to a config.yml, and I'm aware of at least one vendor
> product for flow design & publish via a GUI. Between the
> aforementioned lack of a need to update the agent process, and ongoing
> work [1] to standardize the flow definition for NiFi, Registry,
> Stateless, etc. to VersionedFlowSnapshot, I think MiNiFi should follow
> suit. I can't speak to the C++ side of things but perhaps those more
> familiar with that project can weigh in.
>
> My original email turned out to be more focused on the MVP products,
> but these additional aspects (as well as any others the community
> would like to offer) should be discussed too. If these are not
> necessary for MVP, perhaps we can get the work going on the API and
> MVP and continue the discussions on the Wiki and/or mailing list as
> folks want to contribute the extra goodies :)
>
> Thanks,
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-9069
>
> On Thu, Nov 18, 2021 at 4:07 PM Marc Parisi <ph...@apache.org> wrote:
> >
> > Hi Matt,
> >
> > I think this is great. I like the MVP set of features.
> >
> > Considering use cases and potential maintenance needs, are there thoughts
> > on including RESTART in the MVP for cases where agent UPDATES may be
> > blocked for whatever reason ( I/O, agent problems, etc )?
> >
> > Does the risk of the RESTART verb being somewhat nebulous outweigh any
> > benefits?
> >
> > Again, great stuff!
> >
> > Thanks,
> > Marc
> >
> >
> > On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess <ma...@apache.org>
> wrote:
> >
> > > Hello all,
> > >
> > > I'd like to start a discussion about the current and future Command
> > > and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
> > > codebase. The current implementation[1] is IMO not a pure reference
> > > implementation of the C2 spec [2]. It has fairly limited capability
> > > and uses the PullHttpChangeIngestor for the agent to ask for flow
> > > updates, which makes any bidirectional communication difficult. Rather
> > > than try to build out from what we currently have, I propose a new
> > > implementation (deprecating and eventually removing the old one) that
> > > is more generic, extensible, etc. This would include the following
> > > options:
> > >
> > > - Renaming of the minifi-c2 modules and perhaps the release artifacts,
> > > updating the documentation accordingly
> > > - Creation of a top-level "c2" or "nifi-c2" module that includes an
> > > API module, assembly, NAR, implementation, services bundle, etc. as
> > > needed
> > > - Adding/deleting/changing/using properties related to C2 configuration
> > > - Changing the common runtime/loading code to look for a C2 client
> > > implementation and instantiate that at startup, configuring it with
> > > the appropriate properties and injecting the objects needed for C2
> > > capabilities
> > > - Implementing a new C2 server that supports more of the C2 spec, and
> > > making adding more of the commands easier going forward
> > >
> > > I propose the Minimum Viable Product (MVP) C2 Client would have the
> > > following capabilities:
> > > - A "Manifest Reporter" that can either push a manifest (a document
> > > containing all the agent/instance capabilities such as available
> > > processors and their metadata, e.g.) and/or respond to a DESCRIBE
> > > MANIFEST request (see [2])
> > > - A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply
> > > with ACKNOWLEDGEMENT messages and dispatching any commands such as
> > > UPDATE
> > > - The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in
> > > the old system, but is much more capable
> > > - Supports HTTPS
> > >
> > > I propose the MVP C2 Server would have the following capabilities:
> > > - Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST
> > > - Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially
> > > using the same approach as the old C2 system (loading the flow from a
> > > file for the UPDATE command) but would be extensible/configurable
> > > - Supports HTTPS
> > >
> > > This should give us not only feature parity with the old C2 system,
> > > but also presents a framework to build out the rest of the C2 design
> > > and add other capabilities. I should note that this proposal is not
> > > specific to MiNiFi Java. It should aim to align and use the same
> > > protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2
> > > capabilities to NiFi itself in the future. We could track the proposal
> > > and progress at [4] as I believe that's what the page is intended for.
> > >
> > > I'd like to hear your thoughts on these proposals, and I'm happy to
> > > answer any questions or clarify as necessary. Also as this proceeds
> > > I'm happy to write up any Jira epics/cases/etc. and do the initial
> > > project structure and API setup work so others can dig in.
> > >
> > > Regards,
> > > Matt
> > >
> > > [1]
> https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md
> > > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design
> > > [3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md
> > > [4]
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108
> > >
>

Re: [DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Matt Burgess <ma...@apache.org>]

Marc,

That's a great point and reminds me that I missed two whole sections
of the proposal: 1) The refactor of minifi-bootstrap, and 2) the
standardization of serialized flow control to VersionedFlowSnapshot
(i.e. JSON not XML or YAML).

1) Currently it's the bootstrap process that reaches out to the C2
server and handles the commands such as UPDATE and RESTART. In fact,
UPDATE updates the flow on-disk and RESTART then kills and restarts
the MiNiFi process.  I think with the time that's passed since its
inception, it has kind of gotten bloated and has taken on capabilities
probably best left to a separate module, since those things make it
inconsistent with the way NiFi bootstrap works . In that vein I'd
propose (at some point, not necessarily MVP but that'd be great) that
we standardize the bootstraps as much as possible, and put all the C2
stuff into the C2 modules (i.e. the C2 Client in the case of
minifi-bootstrap). I vaguely alluded to that when I said the C2 client
would need access to the internal objects, but that glossed WAY over
the actual implication, which is that the C2 client module will be the
thing talking to the C2 server to handle all the C2 messages including
UPDATE, and I think RESTART might be replaced with RELOAD or absorbed
into UPDATE, but happy to discuss that as well.

If there are issues such as updating an agent/instance with
configuration properties, I anticipate a new proposal and design to
cover that, but I invite current C2 Server users to jump in here
please. As it stands I believe we haven't implemented enough of the C2
capabilities to be affected, unless the C2 server's flow update file
includes configurable property updates. Otherwise I would think just a
flow update would suffice, and could be done without restarting the
process itself. This would align it more to the way NiFi works as
well, allowing C2 to be used for flow control and metadata exchange
rather than process control.

2) IIRC, the "config.yml' approach came from two use cases, to be able
to update the agent process/configuration itself and to make it easier
to "design" a flow by having more user-friendly names for the various
configuration and component properties without a UI designer. These
days, the minifi-toolkit allows the user to convert a flow from
flow.xml.gz to a config.yml, and I'm aware of at least one vendor
product for flow design & publish via a GUI. Between the
aforementioned lack of a need to update the agent process, and ongoing
work [1] to standardize the flow definition for NiFi, Registry,
Stateless, etc. to VersionedFlowSnapshot, I think MiNiFi should follow
suit. I can't speak to the C++ side of things but perhaps those more
familiar with that project can weigh in.

My original email turned out to be more focused on the MVP products,
but these additional aspects (as well as any others the community
would like to offer) should be discussed too. If these are not
necessary for MVP, perhaps we can get the work going on the API and
MVP and continue the discussions on the Wiki and/or mailing list as
folks want to contribute the extra goodies :)

Thanks,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-9069

On Thu, Nov 18, 2021 at 4:07 PM Marc Parisi <ph...@apache.org> wrote:
>
> Hi Matt,
>
> I think this is great. I like the MVP set of features.
>
> Considering use cases and potential maintenance needs, are there thoughts
> on including RESTART in the MVP for cases where agent UPDATES may be
> blocked for whatever reason ( I/O, agent problems, etc )?
>
> Does the risk of the RESTART verb being somewhat nebulous outweigh any
> benefits?
>
> Again, great stuff!
>
> Thanks,
> Marc
>
>
> On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess <ma...@apache.org> wrote:
>
> > Hello all,
> >
> > I'd like to start a discussion about the current and future Command
> > and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
> > codebase. The current implementation[1] is IMO not a pure reference
> > implementation of the C2 spec [2]. It has fairly limited capability
> > and uses the PullHttpChangeIngestor for the agent to ask for flow
> > updates, which makes any bidirectional communication difficult. Rather
> > than try to build out from what we currently have, I propose a new
> > implementation (deprecating and eventually removing the old one) that
> > is more generic, extensible, etc. This would include the following
> > options:
> >
> > - Renaming of the minifi-c2 modules and perhaps the release artifacts,
> > updating the documentation accordingly
> > - Creation of a top-level "c2" or "nifi-c2" module that includes an
> > API module, assembly, NAR, implementation, services bundle, etc. as
> > needed
> > - Adding/deleting/changing/using properties related to C2 configuration
> > - Changing the common runtime/loading code to look for a C2 client
> > implementation and instantiate that at startup, configuring it with
> > the appropriate properties and injecting the objects needed for C2
> > capabilities
> > - Implementing a new C2 server that supports more of the C2 spec, and
> > making adding more of the commands easier going forward
> >
> > I propose the Minimum Viable Product (MVP) C2 Client would have the
> > following capabilities:
> > - A "Manifest Reporter" that can either push a manifest (a document
> > containing all the agent/instance capabilities such as available
> > processors and their metadata, e.g.) and/or respond to a DESCRIBE
> > MANIFEST request (see [2])
> > - A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply
> > with ACKNOWLEDGEMENT messages and dispatching any commands such as
> > UPDATE
> > - The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in
> > the old system, but is much more capable
> > - Supports HTTPS
> >
> > I propose the MVP C2 Server would have the following capabilities:
> > - Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST
> > - Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially
> > using the same approach as the old C2 system (loading the flow from a
> > file for the UPDATE command) but would be extensible/configurable
> > - Supports HTTPS
> >
> > This should give us not only feature parity with the old C2 system,
> > but also presents a framework to build out the rest of the C2 design
> > and add other capabilities. I should note that this proposal is not
> > specific to MiNiFi Java. It should aim to align and use the same
> > protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2
> > capabilities to NiFi itself in the future. We could track the proposal
> > and progress at [4] as I believe that's what the page is intended for.
> >
> > I'd like to hear your thoughts on these proposals, and I'm happy to
> > answer any questions or clarify as necessary. Also as this proceeds
> > I'm happy to write up any Jira epics/cases/etc. and do the initial
> > project structure and API setup work so others can dig in.
> >
> > Regards,
> > Matt
> >
> > [1] https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md
> > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design
> > [3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md
> > [4]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108
> >

Re: [DISCUSS] The future of (Mi)NiFi Command and Control (C2)

[Marc Parisi <ph...@apache.org>]

Hi Matt,

I think this is great. I like the MVP set of features.

Considering use cases and potential maintenance needs, are there thoughts
on including RESTART in the MVP for cases where agent UPDATES may be
blocked for whatever reason ( I/O, agent problems, etc )?

Does the risk of the RESTART verb being somewhat nebulous outweigh any
benefits?

Again, great stuff!

Thanks,
Marc


On Thu, Nov 18, 2021 at 2:20 PM Matt Burgess <ma...@apache.org> wrote:

> Hello all,
>
> I'd like to start a discussion about the current and future Command
> and Control (C2) capabilities in terms of the NiFi/MiNiFi Java
> codebase. The current implementation[1] is IMO not a pure reference
> implementation of the C2 spec [2]. It has fairly limited capability
> and uses the PullHttpChangeIngestor for the agent to ask for flow
> updates, which makes any bidirectional communication difficult. Rather
> than try to build out from what we currently have, I propose a new
> implementation (deprecating and eventually removing the old one) that
> is more generic, extensible, etc. This would include the following
> options:
>
> - Renaming of the minifi-c2 modules and perhaps the release artifacts,
> updating the documentation accordingly
> - Creation of a top-level "c2" or "nifi-c2" module that includes an
> API module, assembly, NAR, implementation, services bundle, etc. as
> needed
> - Adding/deleting/changing/using properties related to C2 configuration
> - Changing the common runtime/loading code to look for a C2 client
> implementation and instantiate that at startup, configuring it with
> the appropriate properties and injecting the objects needed for C2
> capabilities
> - Implementing a new C2 server that supports more of the C2 spec, and
> making adding more of the commands easier going forward
>
> I propose the Minimum Viable Product (MVP) C2 Client would have the
> following capabilities:
> - A "Manifest Reporter" that can either push a manifest (a document
> containing all the agent/instance capabilities such as available
> processors and their metadata, e.g.) and/or respond to a DESCRIBE
> MANIFEST request (see [2])
> - A "Heartbeat Reporter" to send HEARTBEAT messages and also to reply
> with ACKNOWLEDGEMENT messages and dispatching any commands such as
> UPDATE
> - The Heartbeat Reporter is analogous to the PullHttpChangeIngestor in
> the old system, but is much more capable
> - Supports HTTPS
>
> I propose the MVP C2 Server would have the following capabilities:
> - Handles manifest messages (push and/or pull) aka DESCRIBE MANIFEST
> - Handles the HEARTBEAT / UPDATE / ACK exchange, perhaps initially
> using the same approach as the old C2 system (loading the flow from a
> file for the UPDATE command) but would be extensible/configurable
> - Supports HTTPS
>
> This should give us not only feature parity with the old C2 system,
> but also presents a framework to build out the rest of the C2 design
> and add other capabilities. I should note that this proposal is not
> specific to MiNiFi Java. It should aim to align and use the same
> protocol(s) as MiNiFi C++ [3], and should allow us to perhaps add C2
> capabilities to NiFi itself in the future. We could track the proposal
> and progress at [4] as I believe that's what the page is intended for.
>
> I'd like to hear your thoughts on these proposals, and I'm happy to
> answer any questions or clarify as necessary. Also as this proceeds
> I'm happy to write up any Jira epics/cases/etc. and do the initial
> project structure and API setup work so others can dig in.
>
> Regards,
> Matt
>
> [1] https://github.com/apache/nifi/blob/main/minifi/minifi-c2/README.md
> [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design
> [3] https://github.com/apache/nifi-minifi-cpp/blob/main/C2.md
> [4]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177048430&flashId=624428108
>