You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Aldrin Piri <al...@gmail.com> on 2018/02/28 06:53:27 UTC
[DISCUSS] MiNiFi Flow Designer
Hey folks,
With the release of Registry, I’ve been contemplating leveraging it and the
current codebase of NiFi to facilitate ease of use for MiNiFi flow design.
One of the areas I would like to form a concerted effort around is that of
the Command and Control (C2) functionality originally presented when the
MiNiFi effort was started and further expounded upon with a feature
proposal [1]. In that proposal, while the names are dated, we have
components that fulfill some of the core building blocks toward the overall
vision of C2.
For those not intimately familiar, MiNiFi is primarily configured via a
YAML configuration file. [2] The initial idea was that flows would be
simple a source, maybe a transform or two, and then transmission. The
notion was that hand creation of one of these files would not be overly
onerous. It became apparent from questions, however, that users were doing
more complex flows and this YAML ranged from tedious to difficult for those
situations. This precipitated the best approach currently available,
utilizing a NiFi instance as a way to generate a template. This template
was then exported, and using our provided toolkit to get YAML out that is
deployable with your instance. Better, but far from ideal.
A couple approaches came to mind to further narrow this gap of user
expectation and realities of our configuration.
(1) In our current state, we have heavily relied on convenience of
component overlap. Generally speaking, we relied on the notion that NiFi’s
bundled components are a superset of those found in MiNiFi
implementations. We could have continued that, but this would not
necessarily free the user of the file handling and transformation
processes. Additionally, this comes with caveats, among them that given
the ways in which C++ allows us different accesses, there are some
interesting processors that work better for that more native approach and
might not ever justify having an equivalent in Java.
(2) Providing a first class experience in NiFi for MiNiFi flow designing
making use of the existing UI components as a base and a new design mode
specifically for providing a clean user experience, letting users operate
in a different context but a familiar environment. Certainly preferred,
but how this fit was not clear.
As a crash course in MiNiFi efforts, there have been some key contributions
that I believe when more strongly united can generate a very complete
authoring and flow update process. Some of these items are:
- C2 API Protocol specification and implementation in C++ [3]
- C2 Server with the notion of serving different flows [4] backed by
varying implementations, one of which was via REST to access NiFi Templates
directly [5]
- Warm Redeploy in MiNiFi Java [6]
One of the first questions to come to mind was, “Is within NiFi the right
place?”
I think there are several reasons that this is reasonable. Seeing some of
the discussion around the consolidation of common UI components presented
by Scott Aslan, I think sharing the core of the UI elements between the
traditional NiFi and MiNiFi flow designs is a logical extension. While
those flows in MiNiFi are not real-time changes we are dealing with the
same design elements in creating directed graphs. Additionally, while
MiNiFi instances will likely not ever have the same breadth of extensions
supported, there is nothing inherently prohibitive of a more complex graph
comprised of larger numbers of edges and/or vertices. In my view, the
mechanics are the same but just being performed in a different contexts.
In the case of MiNiFi, this is effectively an “offline” context. By
"offline," I mean we are not strictly bound to loaded components such as
that in the current NiFi context consisting of items from unpacked NARs.
How can we minimize the impact to NiFi?
Given that this is a new capability, I don’t know that we necessarily need
to interfere with the existing APIs and could possibly also make this an
optional feature. From this standpoint, we could develop new endpoints for
this specific context. Some of this work would rely on external services
and/or definitions to drive the different context. For example, without
NARs acting as the source of available components, something like the C2
server could provide extensions for MiNiFi instances it is aware of to help
drive the experience. This data would be a skeletal view of a component
definition/interface without any of the inherent code backing it needed.
How do we maximize the utility of these efforts?
With the idea about powering MiNiFi flow design with specific sets of
components, I could see this extending to even NiFi itself. Consider the
use case where something like I wish to make use of additional artifacts
along the lines of [7] and an extension of the current Registry
implementation. As a simple example, with an understanding of what the
universe of components is available to my system and not just those
currently loaded, it would be possible for users to design for other
instances. With these hooks in place, there are a few permutations of this
relative concept that enable us to carry out some of the “nice to haves” we
haven’t quite been able to address.
How does this all come together for MiNiFi?
I have created a work of sheer artistic mastery on the Command and Control
proposal to illustrate a rough sequence and architecture. [8] This diagram
shows from entering MiNiFi flow design through to an instance updating to
the new flow.
Overall, Registry was a tremendous step forward for the NiFi ecosystem. I
think in our current state, we have reached a convergence of capabilities
that allow us to build higher level abstractions for dataflow management.
I think the approach of (2) is a great first foray into that realm that
provides an immediate solution for a gap in our MiNiFi efforts.
Additionally, this approach lays the foundation for additional applications
possibly inclusive of core NiFi.
I look forward to your thoughts and commentary. Certainly many details to
think and reason about, but I see some incredible potential with the work
we've done thus far both near and long term.
Thanks for your time!
--aldrin
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Andy Christianson <ai...@protonmail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Great discussion on this topic.
Something I'd like to add which I haven't seen yet, is a MiNiFi production
environment emulator, perhaps talking to a virtualization system like QEMU, to
be used during the flow design & development process. The purpose of this would
be to provide some level of verification for an Edge/IoT flow before it is
deployed out to devices. The minifi binary used in the emulation environment
would hopefully be the same binary in the production environment, so at least
we can verify that the config.yml produced is compatible with the minifi binary
used.
I'm not sure whether this should be integrated into the design/development UI
or whether it should be a separate tool that can be used as part of the process
and potentially integrated with CI/CD systems.
Regards,
Andy I.C.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQEcBAEBAgAGBQJanVOOAAoJEG1+mBKNMpIDV+QIAI3Ii1OQs5D2oOydhcfR765I
cbuRTQZc0uiWhrPrdppyqaUkiSmpuW3Z/mi+vM+tbLIVrheMF0EZ7WQ1DKWKwdZJ
r2I/0CyQ5mkhWRLZDjz4YyKsWXJznrubSz4O2rExjCENiMCsPT/OU3Qpfu7HH9nE
ug0aD17q/KGxLLv8WnT8xlW74PSOvHRdtuGI7kiuZtf9NqSY+V0M8uxp71T9UCuS
j7xDeApCRlYsDuKxc7tg3qdnIROOPhfxKqf7nP6Miymt6cThiGSDSlII01RnkFWP
0VK/S3EWuWAoAe8L31W/Mzdfh+WmzUqLrFsmqmO9yz0Y6nswKecAV8setGL0p9Q=
=BCWa
-----END PGP SIGNATURE-----
Sent from ProtonMail, Swiss-based encrypted email.
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Aldrin Piri <al...@gmail.com>.
Hey folks,
Thanks for all the input and feedback thus far. Going to try to address
some points that came up that I have some thoughts on.
Andy,
As for continued YAML support, it is the intent to do so moving forward.
In some respects, at least for initial flow authoring/deployment
capabilities, I think it is a fundamental part where the C2 server would
help to translate the Versioned Flow content into a YAML format. As we get
more sophisticated we could have other, more efficient/compact
representations in addition.
Scotty,
In terms of the UI/UX work with the subproject being discussed, it was less
about using those components directly and more of an analog where we are
consolidating efforts into one place. The comparison I was trying to make
was that whatever the UI/UX is, new or old, has some very strong
similarities between both NiFi and MiNiFi with slightly different
contexts. Apologies for the lack of clarity on that front.
Andrew,
Great thoughts and I think your concept of the groups with the available
extensions is key. With such hooks in place, the applicability of the
general workflow and tooling could have some broader applications to other
efforts as has been suggested by some of the other folks in this chain.
Knowing that components exist powered by something like previously
discussed Extension handling in Registry, even if they are not currently on
"this" instance, can open up some really interesting possibilities in terms
of generating assemblies and other party tricks.
Thanks for your thoughts and time!
--aldrin
On Thu, Mar 1, 2018 at 8:12 PM, Andy LoPresto <al...@apache.org> wrote:
> I wonder if extending this model to all *NiFi flows makes sense. A common
> user request we have seen is to provide for a “flow development”
> environment separate from the “flow execution” environment (this was one of
> the leading impetuses for the FDLC enhancement effort that led to the
> Registry). Some users expect to allow for a priori flow development, which
> is then promoted as a static flow definition to a disparate production-like
> environment where it can run and be monitored but not be editable.
>
> I think extending the NiFi Flow Designer + MiNiFi “flow execution”
> scenario to n-many execution environments (MiNiFi - Java, MiNiFi - C++,
> NiFi “headless”/“run-only”) might make sense here. We already have code to
> export the same flow (object graph) to YAML, XML, etc., so the “Design”
> portion could be on a specific server/network/etc. and have specific user
> access and permissions, and could be connected to a Registry instance, so
> that when a flow is deemed “complete” it is exported/saved/checked-in to
> the Registry, and then one or more “execution environments” (MiNiFi - *,
> a regular NiFi node, a “monitor-only” NiFi node) could all check-out the
> flow in the appropriate format and import/run that flow.
>
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Mar 1, 2018, at 4:39 PM, Andrew Grande <ap...@gmail.com> wrote:
>
> My 2c.
>
> Using NiFi registry for both NiFi and MiNiFi flows is ideal. For the
> designer, IMO, adding another process/service to host it is not too great,
> as it adds to the complexity of overall ecosystem (the nifi/minifi java/cpp
> is already quite involved with c2 server, etc). I would like to see a
> familiar NiFi UI concept evolve to have a 'profile' e.g. on a group level.
> This profile would dictate which processors are available for that
> particular runtime. The same profile eould be associated with a flow in the
> registry.
>
> The main concept is to stay within the same familiar design environment for
> all deployment targets, but make NiFi understand what set of available
> components and behaviors are available for the current target at hand.
>
> Andrew
>
> On Thu, Mar 1, 2018, 7:30 PM Jeff Zemerick <jz...@apache.org> wrote:
>
> I think it sounds like a good idea. The majority of the MiNiFi flows I have
> made are all fairly simple and I have not found the YAML authoring to be
> overly burdensome. The more complex flows were done in NiFi and converted.
> So as long as the option to make the flows by hand remains (and I don't
> think there was anything in the proposal to the contrary), I think it would
> be a beneficial addition.
>
> Jeff
>
>
> On Thu, Mar 1, 2018 at 2:21 PM, Scott Aslan <sc...@gmail.com> wrote:
>
> AndyC/Aldrin,
>
> The UI/UX design system sub project currently under discussion will be
> based off of the work completed during the NiFi Registry and will not
> contain any UI/UX functionality from NiFi. The long term goal would be to
> upgrade angular in the NiFi UI in order to leverage this new design
>
> system.
>
>
> -Scotty
>
>
> On Thu, Mar 1, 2018 at 1:45 PM, Kevin Doran <kd...@apache.org> wrote:
>
> Aldrin,
>
> Thanks a lot for taking the time to write up such a detailed and well
> thought out proposal.
>
> I really like the idea of publishing flows to NiFi Registry that then
>
> get
>
> deployed to MiNiFi agents via the C2 Server. I don't have much to add
> there, and I think that you've identified one of the big missing parts
>
> to
>
> make that experience user-friendly -- how to design and author the
>
> flows
>
> in
>
> the first place.
>
> The MiNiFi Flow Designer would fill that need in a really elegant way
>
> with
>
> a low learning curve for NiFi users. Having the ability to design and
> author flows in an environment other than the "runtime" environment one
>
> is
>
> targeting would be a huge step forward. Not just for MiNiFi, but
> multi-environment deployments of NiFi, where one might be developing a
>
> flow
>
> in an environment other than where they intend to run it. This would
>
> make
>
> the SDLC options for NiFi flows even more flexible, which is a big part
>
> of
>
> the goal of the new NiFi Registry.
>
> It certainly presents some challenges, but I definitely think the
>
> upside
>
> for MiNiFi flow authoring would be huge.
>
> Thanks,
> Kevin
>
> On 2/28/18, 07:53, "Aldrin Piri" <al...@gmail.com> wrote:
>
> It appears I accidentally left off the actual links. Here they are
>
> in
>
> all
> their glory:
>
> [1]
> https://cwiki.apache.org/confluence/display/MINIFI/
> MiNiFi+Command+and+Control
> [2]
> https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
> [3] https://issues.apache.org/jira/browse/MINIFICPP-36
> [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
> [5]
> https://github.com/apache/nifi-minifi/tree/master/
> minifi-c2#configuration-providers
> [6]
> https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#
> automatic-warm-redeploy
> [7]
> https://cwiki.apache.org/confluence/display/NIFI/
> Extension+Repositories+%28aka+Extension+Registry%29+for+
> Dynamically-loaded+Extensions
> [8]
> https://cwiki.apache.org/confluence/display/MINIFI/
>
> MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
>
>
>
> On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinpiri@gmail.com
>
>
> wrote:
>
> Hey folks,
>
> With the release of Registry, I’ve been contemplating leveraging
>
> it
>
> and
>
> the current codebase of NiFi to facilitate ease of use for MiNiFi
>
> flow
>
> design. One of the areas I would like to form a concerted effort
>
> around is
>
> that of the Command and Control (C2) functionality originally
>
> presented
>
> when the MiNiFi effort was started and further expounded upon
>
> with
>
> a
>
> feature proposal [1]. In that proposal, while the names are
>
> dated,
>
> we have
>
> components that fulfill some of the core building blocks toward
>
> the
>
> overall
>
> vision of C2.
>
> For those not intimately familiar, MiNiFi is primarily configured
>
> via a
>
> YAML configuration file. [2] The initial idea was that flows
>
> would
>
> be
>
> simple a source, maybe a transform or two, and then transmission.
>
> The
>
> notion was that hand creation of one of these files would not be
>
> overly
>
> onerous. It became apparent from questions, however, that users
>
> were doing
>
> more complex flows and this YAML ranged from tedious to difficult
>
> for those
>
> situations. This precipitated the best approach currently
>
> available,
>
> utilizing a NiFi instance as a way to generate a template. This
>
> template
>
> was then exported, and using our provided toolkit to get YAML out
>
> that is
>
> deployable with your instance. Better, but far from ideal.
>
> A couple approaches came to mind to further narrow this gap of
>
> user
>
> expectation and realities of our configuration.
>
> (1) In our current state, we have heavily relied on convenience
>
> of
>
> component overlap. Generally speaking, we relied on the notion
>
> that
>
> NiFi’s
>
> bundled components are a superset of those found in MiNiFi
> implementations. We could have continued that, but this would
>
> not
>
> necessarily free the user of the file handling and transformation
> processes. Additionally, this comes with caveats, among them
>
> that
>
> given
>
> the ways in which C++ allows us different accesses, there are
>
> some
>
> interesting processors that work better for that more native
>
> approach and
>
> might not ever justify having an equivalent in Java.
>
> (2) Providing a first class experience in NiFi for MiNiFi flow
>
> designing
>
> making use of the existing UI components as a base and a new
>
> design
>
> mode
>
> specifically for providing a clean user experience, letting users
>
> operate
>
> in a different context but a familiar environment. Certainly
>
> preferred,
>
> but how this fit was not clear.
>
> As a crash course in MiNiFi efforts, there have been some key
> contributions that I believe when more strongly united can
>
> generate
>
> a very
>
> complete authoring and flow update process. Some of these items
>
> are:
>
>
> - C2 API Protocol specification and implementation in C++ [3]
> - C2 Server with the notion of serving different flows [4]
>
> backed
>
> by
>
> varying implementations, one of which was via REST to access
>
> NiFi
>
> Templates
>
> directly [5]
> - Warm Redeploy in MiNiFi Java [6]
>
>
> One of the first questions to come to mind was, “Is within NiFi
>
> the
>
> right
>
> place?”
>
> I think there are several reasons that this is reasonable. Seeing
>
> some of
>
> the discussion around the consolidation of common UI components
>
> presented
>
> by Scott Aslan, I think sharing the core of the UI elements
>
> between
>
> the
>
> traditional NiFi and MiNiFi flow designs is a logical extension.
>
> While
>
> those flows in MiNiFi are not real-time changes we are dealing
>
> with
>
> the
>
> same design elements in creating directed graphs. Additionally,
>
> while
>
> MiNiFi instances will likely not ever have the same breadth of
>
> extensions
>
> supported, there is nothing inherently prohibitive of a more
>
> complex
>
> graph
>
> comprised of larger numbers of edges and/or vertices. In my
>
> view,
>
> the
>
> mechanics are the same but just being performed in a different
>
> contexts.
>
> In the case of MiNiFi, this is effectively an “offline” context.
>
> By
>
> "offline," I mean we are not strictly bound to loaded components
>
> such as
>
> that in the current NiFi context consisting of items from
>
> unpacked
>
> NARs.
>
>
> How can we minimize the impact to NiFi?
>
> Given that this is a new capability, I don’t know that we
>
> necessarily need
>
> to interfere with the existing APIs and could possibly also make
>
> this an
>
> optional feature. From this standpoint, we could develop new
>
> endpoints for
>
> this specific context. Some of this work would rely on external
>
> services
>
> and/or definitions to drive the different context. For example,
>
> without
>
> NARs acting as the source of available components, something like
>
> the C2
>
> server could provide extensions for MiNiFi instances it is aware
>
> of
>
> to help
>
> drive the experience. This data would be a skeletal view of a
>
> component
>
> definition/interface without any of the inherent code backing it
>
> needed.
>
>
> How do we maximize the utility of these efforts?
>
> With the idea about powering MiNiFi flow design with specific
>
> sets
>
> of
>
> components, I could see this extending to even NiFi itself.
>
> Consider the
>
> use case where something like I wish to make use of additional
>
> artifacts
>
> along the lines of [7] and an extension of the current Registry
> implementation. As a simple example, with an understanding of
>
> what
>
> the
>
> universe of components is available to my system and not just
>
> those
>
> currently loaded, it would be possible for users to design for
>
> other
>
> instances. With these hooks in place, there are a few
>
> permutations
>
> of this
>
> relative concept that enable us to carry out some of the “nice to
>
> haves” we
>
> haven’t quite been able to address.
>
> How does this all come together for MiNiFi?
>
> I have created a work of sheer artistic mastery on the Command
>
> and
>
> Control
>
> proposal to illustrate a rough sequence and architecture. [8]
>
> This
>
> diagram
>
> shows from entering MiNiFi flow design through to an instance
>
> updating to
>
> the new flow.
>
>
> Overall, Registry was a tremendous step forward for the NiFi
>
> ecosystem. I
>
> think in our current state, we have reached a convergence of
>
> capabilities
>
> that allow us to build higher level abstractions for dataflow
>
> management.
>
> I think the approach of (2) is a great first foray into that
>
> realm
>
> that
>
> provides an immediate solution for a gap in our MiNiFi efforts.
> Additionally, this approach lays the foundation for additional
>
> applications
>
> possibly inclusive of core NiFi.
>
> I look forward to your thoughts and commentary. Certainly many
>
> details to
>
> think and reason about, but I see some incredible potential with
>
> the
>
> work
>
> we've done thus far both near and long term.
>
> Thanks for your time!
> --aldrin
>
>
>
>
>
>
>
>
>
>
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Andy LoPresto <al...@apache.org>.
I wonder if extending this model to all *NiFi flows makes sense. A common user request we have seen is to provide for a “flow development” environment separate from the “flow execution” environment (this was one of the leading impetuses for the FDLC enhancement effort that led to the Registry). Some users expect to allow for a priori flow development, which is then promoted as a static flow definition to a disparate production-like environment where it can run and be monitored but not be editable.
I think extending the NiFi Flow Designer + MiNiFi “flow execution” scenario to n-many execution environments (MiNiFi - Java, MiNiFi - C++, NiFi “headless”/“run-only”) might make sense here. We already have code to export the same flow (object graph) to YAML, XML, etc., so the “Design” portion could be on a specific server/network/etc. and have specific user access and permissions, and could be connected to a Registry instance, so that when a flow is deemed “complete” it is exported/saved/checked-in to the Registry, and then one or more “execution environments” (MiNiFi - *, a regular NiFi node, a “monitor-only” NiFi node) could all check-out the flow in the appropriate format and import/run that flow.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Mar 1, 2018, at 4:39 PM, Andrew Grande <ap...@gmail.com> wrote:
>
> My 2c.
>
> Using NiFi registry for both NiFi and MiNiFi flows is ideal. For the
> designer, IMO, adding another process/service to host it is not too great,
> as it adds to the complexity of overall ecosystem (the nifi/minifi java/cpp
> is already quite involved with c2 server, etc). I would like to see a
> familiar NiFi UI concept evolve to have a 'profile' e.g. on a group level.
> This profile would dictate which processors are available for that
> particular runtime. The same profile eould be associated with a flow in the
> registry.
>
> The main concept is to stay within the same familiar design environment for
> all deployment targets, but make NiFi understand what set of available
> components and behaviors are available for the current target at hand.
>
> Andrew
>
> On Thu, Mar 1, 2018, 7:30 PM Jeff Zemerick <jz...@apache.org> wrote:
>
>> I think it sounds like a good idea. The majority of the MiNiFi flows I have
>> made are all fairly simple and I have not found the YAML authoring to be
>> overly burdensome. The more complex flows were done in NiFi and converted.
>> So as long as the option to make the flows by hand remains (and I don't
>> think there was anything in the proposal to the contrary), I think it would
>> be a beneficial addition.
>>
>> Jeff
>>
>>
>> On Thu, Mar 1, 2018 at 2:21 PM, Scott Aslan <sc...@gmail.com> wrote:
>>
>>> AndyC/Aldrin,
>>>
>>> The UI/UX design system sub project currently under discussion will be
>>> based off of the work completed during the NiFi Registry and will not
>>> contain any UI/UX functionality from NiFi. The long term goal would be to
>>> upgrade angular in the NiFi UI in order to leverage this new design
>> system.
>>>
>>> -Scotty
>>>
>>>
>>> On Thu, Mar 1, 2018 at 1:45 PM, Kevin Doran <kd...@apache.org> wrote:
>>>
>>>> Aldrin,
>>>>
>>>> Thanks a lot for taking the time to write up such a detailed and well
>>>> thought out proposal.
>>>>
>>>> I really like the idea of publishing flows to NiFi Registry that then
>> get
>>>> deployed to MiNiFi agents via the C2 Server. I don't have much to add
>>>> there, and I think that you've identified one of the big missing parts
>> to
>>>> make that experience user-friendly -- how to design and author the
>> flows
>>> in
>>>> the first place.
>>>>
>>>> The MiNiFi Flow Designer would fill that need in a really elegant way
>>> with
>>>> a low learning curve for NiFi users. Having the ability to design and
>>>> author flows in an environment other than the "runtime" environment one
>>> is
>>>> targeting would be a huge step forward. Not just for MiNiFi, but
>>>> multi-environment deployments of NiFi, where one might be developing a
>>> flow
>>>> in an environment other than where they intend to run it. This would
>> make
>>>> the SDLC options for NiFi flows even more flexible, which is a big part
>>> of
>>>> the goal of the new NiFi Registry.
>>>>
>>>> It certainly presents some challenges, but I definitely think the
>> upside
>>>> for MiNiFi flow authoring would be huge.
>>>>
>>>> Thanks,
>>>> Kevin
>>>>
>>>> On 2/28/18, 07:53, "Aldrin Piri" <al...@gmail.com> wrote:
>>>>
>>>> It appears I accidentally left off the actual links. Here they are
>>> in
>>>> all
>>>> their glory:
>>>>
>>>> [1]
>>>> https://cwiki.apache.org/confluence/display/MINIFI/
>>>> MiNiFi+Command+and+Control
>>>> [2]
>>>> https://github.com/apache/nifi-minifi/blob/master/
>>>> minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
>>>> [3] https://issues.apache.org/jira/browse/MINIFICPP-36
>>>> [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
>>>> [5]
>>>> https://github.com/apache/nifi-minifi/tree/master/
>>>> minifi-c2#configuration-providers
>>>> [6]
>>>> https://github.com/apache/nifi-minifi/blob/master/
>>>> minifi-docs/src/main/markdown/System_Admin_Guide.md#
>>>> automatic-warm-redeploy
>>>> [7]
>>>> https://cwiki.apache.org/confluence/display/NIFI/
>>>> Extension+Repositories+%28aka+Extension+Registry%29+for+
>>>> Dynamically-loaded+Extensions
>>>> [8]
>>>> https://cwiki.apache.org/confluence/display/MINIFI/
>>>>
>> MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
>>>>
>>>>
>>>> On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinpiri@gmail.com
>>>
>>>> wrote:
>>>>
>>>>> Hey folks,
>>>>>
>>>>> With the release of Registry, I’ve been contemplating leveraging
>> it
>>>> and
>>>>> the current codebase of NiFi to facilitate ease of use for MiNiFi
>>>> flow
>>>>> design. One of the areas I would like to form a concerted effort
>>>> around is
>>>>> that of the Command and Control (C2) functionality originally
>>>> presented
>>>>> when the MiNiFi effort was started and further expounded upon
>> with
>>> a
>>>>> feature proposal [1]. In that proposal, while the names are
>> dated,
>>>> we have
>>>>> components that fulfill some of the core building blocks toward
>> the
>>>> overall
>>>>> vision of C2.
>>>>>
>>>>> For those not intimately familiar, MiNiFi is primarily configured
>>>> via a
>>>>> YAML configuration file. [2] The initial idea was that flows
>>> would
>>>> be
>>>>> simple a source, maybe a transform or two, and then transmission.
>>>> The
>>>>> notion was that hand creation of one of these files would not be
>>>> overly
>>>>> onerous. It became apparent from questions, however, that users
>>>> were doing
>>>>> more complex flows and this YAML ranged from tedious to difficult
>>>> for those
>>>>> situations. This precipitated the best approach currently
>>> available,
>>>>> utilizing a NiFi instance as a way to generate a template. This
>>>> template
>>>>> was then exported, and using our provided toolkit to get YAML out
>>>> that is
>>>>> deployable with your instance. Better, but far from ideal.
>>>>>
>>>>> A couple approaches came to mind to further narrow this gap of
>> user
>>>>> expectation and realities of our configuration.
>>>>>
>>>>> (1) In our current state, we have heavily relied on convenience
>> of
>>>>> component overlap. Generally speaking, we relied on the notion
>>> that
>>>> NiFi’s
>>>>> bundled components are a superset of those found in MiNiFi
>>>>> implementations. We could have continued that, but this would
>> not
>>>>> necessarily free the user of the file handling and transformation
>>>>> processes. Additionally, this comes with caveats, among them
>> that
>>>> given
>>>>> the ways in which C++ allows us different accesses, there are
>> some
>>>>> interesting processors that work better for that more native
>>>> approach and
>>>>> might not ever justify having an equivalent in Java.
>>>>>
>>>>> (2) Providing a first class experience in NiFi for MiNiFi flow
>>>> designing
>>>>> making use of the existing UI components as a base and a new
>> design
>>>> mode
>>>>> specifically for providing a clean user experience, letting users
>>>> operate
>>>>> in a different context but a familiar environment. Certainly
>>>> preferred,
>>>>> but how this fit was not clear.
>>>>>
>>>>> As a crash course in MiNiFi efforts, there have been some key
>>>>> contributions that I believe when more strongly united can
>> generate
>>>> a very
>>>>> complete authoring and flow update process. Some of these items
>>> are:
>>>>>
>>>>> - C2 API Protocol specification and implementation in C++ [3]
>>>>> - C2 Server with the notion of serving different flows [4]
>>> backed
>>>> by
>>>>> varying implementations, one of which was via REST to access
>>> NiFi
>>>> Templates
>>>>> directly [5]
>>>>> - Warm Redeploy in MiNiFi Java [6]
>>>>>
>>>>>
>>>>> One of the first questions to come to mind was, “Is within NiFi
>> the
>>>> right
>>>>> place?”
>>>>>
>>>>> I think there are several reasons that this is reasonable. Seeing
>>>> some of
>>>>> the discussion around the consolidation of common UI components
>>>> presented
>>>>> by Scott Aslan, I think sharing the core of the UI elements
>> between
>>>> the
>>>>> traditional NiFi and MiNiFi flow designs is a logical extension.
>>>> While
>>>>> those flows in MiNiFi are not real-time changes we are dealing
>> with
>>>> the
>>>>> same design elements in creating directed graphs. Additionally,
>>>> while
>>>>> MiNiFi instances will likely not ever have the same breadth of
>>>> extensions
>>>>> supported, there is nothing inherently prohibitive of a more
>>> complex
>>>> graph
>>>>> comprised of larger numbers of edges and/or vertices. In my
>> view,
>>>> the
>>>>> mechanics are the same but just being performed in a different
>>>> contexts.
>>>>> In the case of MiNiFi, this is effectively an “offline” context.
>>> By
>>>>> "offline," I mean we are not strictly bound to loaded components
>>>> such as
>>>>> that in the current NiFi context consisting of items from
>> unpacked
>>>> NARs.
>>>>>
>>>>> How can we minimize the impact to NiFi?
>>>>>
>>>>> Given that this is a new capability, I don’t know that we
>>>> necessarily need
>>>>> to interfere with the existing APIs and could possibly also make
>>>> this an
>>>>> optional feature. From this standpoint, we could develop new
>>>> endpoints for
>>>>> this specific context. Some of this work would rely on external
>>>> services
>>>>> and/or definitions to drive the different context. For example,
>>>> without
>>>>> NARs acting as the source of available components, something like
>>>> the C2
>>>>> server could provide extensions for MiNiFi instances it is aware
>> of
>>>> to help
>>>>> drive the experience. This data would be a skeletal view of a
>>>> component
>>>>> definition/interface without any of the inherent code backing it
>>>> needed.
>>>>>
>>>>> How do we maximize the utility of these efforts?
>>>>>
>>>>> With the idea about powering MiNiFi flow design with specific
>> sets
>>> of
>>>>> components, I could see this extending to even NiFi itself.
>>>> Consider the
>>>>> use case where something like I wish to make use of additional
>>>> artifacts
>>>>> along the lines of [7] and an extension of the current Registry
>>>>> implementation. As a simple example, with an understanding of
>> what
>>>> the
>>>>> universe of components is available to my system and not just
>> those
>>>>> currently loaded, it would be possible for users to design for
>>> other
>>>>> instances. With these hooks in place, there are a few
>> permutations
>>>> of this
>>>>> relative concept that enable us to carry out some of the “nice to
>>>> haves” we
>>>>> haven’t quite been able to address.
>>>>>
>>>>> How does this all come together for MiNiFi?
>>>>>
>>>>> I have created a work of sheer artistic mastery on the Command
>> and
>>>> Control
>>>>> proposal to illustrate a rough sequence and architecture. [8]
>> This
>>>> diagram
>>>>> shows from entering MiNiFi flow design through to an instance
>>>> updating to
>>>>> the new flow.
>>>>>
>>>>>
>>>>> Overall, Registry was a tremendous step forward for the NiFi
>>>> ecosystem. I
>>>>> think in our current state, we have reached a convergence of
>>>> capabilities
>>>>> that allow us to build higher level abstractions for dataflow
>>>> management.
>>>>> I think the approach of (2) is a great first foray into that
>> realm
>>>> that
>>>>> provides an immediate solution for a gap in our MiNiFi efforts.
>>>>> Additionally, this approach lays the foundation for additional
>>>> applications
>>>>> possibly inclusive of core NiFi.
>>>>>
>>>>> I look forward to your thoughts and commentary. Certainly many
>>>> details to
>>>>> think and reason about, but I see some incredible potential with
>>> the
>>>> work
>>>>> we've done thus far both near and long term.
>>>>>
>>>>> Thanks for your time!
>>>>> --aldrin
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Andrew Grande <ap...@gmail.com>.
My 2c.
Using NiFi registry for both NiFi and MiNiFi flows is ideal. For the
designer, IMO, adding another process/service to host it is not too great,
as it adds to the complexity of overall ecosystem (the nifi/minifi java/cpp
is already quite involved with c2 server, etc). I would like to see a
familiar NiFi UI concept evolve to have a 'profile' e.g. on a group level.
This profile would dictate which processors are available for that
particular runtime. The same profile eould be associated with a flow in the
registry.
The main concept is to stay within the same familiar design environment for
all deployment targets, but make NiFi understand what set of available
components and behaviors are available for the current target at hand.
Andrew
On Thu, Mar 1, 2018, 7:30 PM Jeff Zemerick <jz...@apache.org> wrote:
> I think it sounds like a good idea. The majority of the MiNiFi flows I have
> made are all fairly simple and I have not found the YAML authoring to be
> overly burdensome. The more complex flows were done in NiFi and converted.
> So as long as the option to make the flows by hand remains (and I don't
> think there was anything in the proposal to the contrary), I think it would
> be a beneficial addition.
>
> Jeff
>
>
> On Thu, Mar 1, 2018 at 2:21 PM, Scott Aslan <sc...@gmail.com> wrote:
>
> > AndyC/Aldrin,
> >
> > The UI/UX design system sub project currently under discussion will be
> > based off of the work completed during the NiFi Registry and will not
> > contain any UI/UX functionality from NiFi. The long term goal would be to
> > upgrade angular in the NiFi UI in order to leverage this new design
> system.
> >
> > -Scotty
> >
> >
> > On Thu, Mar 1, 2018 at 1:45 PM, Kevin Doran <kd...@apache.org> wrote:
> >
> > > Aldrin,
> > >
> > > Thanks a lot for taking the time to write up such a detailed and well
> > > thought out proposal.
> > >
> > > I really like the idea of publishing flows to NiFi Registry that then
> get
> > > deployed to MiNiFi agents via the C2 Server. I don't have much to add
> > > there, and I think that you've identified one of the big missing parts
> to
> > > make that experience user-friendly -- how to design and author the
> flows
> > in
> > > the first place.
> > >
> > > The MiNiFi Flow Designer would fill that need in a really elegant way
> > with
> > > a low learning curve for NiFi users. Having the ability to design and
> > > author flows in an environment other than the "runtime" environment one
> > is
> > > targeting would be a huge step forward. Not just for MiNiFi, but
> > > multi-environment deployments of NiFi, where one might be developing a
> > flow
> > > in an environment other than where they intend to run it. This would
> make
> > > the SDLC options for NiFi flows even more flexible, which is a big part
> > of
> > > the goal of the new NiFi Registry.
> > >
> > > It certainly presents some challenges, but I definitely think the
> upside
> > > for MiNiFi flow authoring would be huge.
> > >
> > > Thanks,
> > > Kevin
> > >
> > > On 2/28/18, 07:53, "Aldrin Piri" <al...@gmail.com> wrote:
> > >
> > > It appears I accidentally left off the actual links. Here they are
> > in
> > > all
> > > their glory:
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/display/MINIFI/
> > > MiNiFi+Command+and+Control
> > > [2]
> > > https://github.com/apache/nifi-minifi/blob/master/
> > > minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
> > > [3] https://issues.apache.org/jira/browse/MINIFICPP-36
> > > [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
> > > [5]
> > > https://github.com/apache/nifi-minifi/tree/master/
> > > minifi-c2#configuration-providers
> > > [6]
> > > https://github.com/apache/nifi-minifi/blob/master/
> > > minifi-docs/src/main/markdown/System_Admin_Guide.md#
> > > automatic-warm-redeploy
> > > [7]
> > > https://cwiki.apache.org/confluence/display/NIFI/
> > > Extension+Repositories+%28aka+Extension+Registry%29+for+
> > > Dynamically-loaded+Extensions
> > > [8]
> > > https://cwiki.apache.org/confluence/display/MINIFI/
> > >
> MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
> > >
> > >
> > > On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <aldrinpiri@gmail.com
> >
> > > wrote:
> > >
> > > > Hey folks,
> > > >
> > > > With the release of Registry, I’ve been contemplating leveraging
> it
> > > and
> > > > the current codebase of NiFi to facilitate ease of use for MiNiFi
> > > flow
> > > > design. One of the areas I would like to form a concerted effort
> > > around is
> > > > that of the Command and Control (C2) functionality originally
> > > presented
> > > > when the MiNiFi effort was started and further expounded upon
> with
> > a
> > > > feature proposal [1]. In that proposal, while the names are
> dated,
> > > we have
> > > > components that fulfill some of the core building blocks toward
> the
> > > overall
> > > > vision of C2.
> > > >
> > > > For those not intimately familiar, MiNiFi is primarily configured
> > > via a
> > > > YAML configuration file. [2] The initial idea was that flows
> > would
> > > be
> > > > simple a source, maybe a transform or two, and then transmission.
> > > The
> > > > notion was that hand creation of one of these files would not be
> > > overly
> > > > onerous. It became apparent from questions, however, that users
> > > were doing
> > > > more complex flows and this YAML ranged from tedious to difficult
> > > for those
> > > > situations. This precipitated the best approach currently
> > available,
> > > > utilizing a NiFi instance as a way to generate a template. This
> > > template
> > > > was then exported, and using our provided toolkit to get YAML out
> > > that is
> > > > deployable with your instance. Better, but far from ideal.
> > > >
> > > > A couple approaches came to mind to further narrow this gap of
> user
> > > > expectation and realities of our configuration.
> > > >
> > > > (1) In our current state, we have heavily relied on convenience
> of
> > > > component overlap. Generally speaking, we relied on the notion
> > that
> > > NiFi’s
> > > > bundled components are a superset of those found in MiNiFi
> > > > implementations. We could have continued that, but this would
> not
> > > > necessarily free the user of the file handling and transformation
> > > > processes. Additionally, this comes with caveats, among them
> that
> > > given
> > > > the ways in which C++ allows us different accesses, there are
> some
> > > > interesting processors that work better for that more native
> > > approach and
> > > > might not ever justify having an equivalent in Java.
> > > >
> > > > (2) Providing a first class experience in NiFi for MiNiFi flow
> > > designing
> > > > making use of the existing UI components as a base and a new
> design
> > > mode
> > > > specifically for providing a clean user experience, letting users
> > > operate
> > > > in a different context but a familiar environment. Certainly
> > > preferred,
> > > > but how this fit was not clear.
> > > >
> > > > As a crash course in MiNiFi efforts, there have been some key
> > > > contributions that I believe when more strongly united can
> generate
> > > a very
> > > > complete authoring and flow update process. Some of these items
> > are:
> > > >
> > > > - C2 API Protocol specification and implementation in C++ [3]
> > > > - C2 Server with the notion of serving different flows [4]
> > backed
> > > by
> > > > varying implementations, one of which was via REST to access
> > NiFi
> > > Templates
> > > > directly [5]
> > > > - Warm Redeploy in MiNiFi Java [6]
> > > >
> > > >
> > > > One of the first questions to come to mind was, “Is within NiFi
> the
> > > right
> > > > place?”
> > > >
> > > > I think there are several reasons that this is reasonable. Seeing
> > > some of
> > > > the discussion around the consolidation of common UI components
> > > presented
> > > > by Scott Aslan, I think sharing the core of the UI elements
> between
> > > the
> > > > traditional NiFi and MiNiFi flow designs is a logical extension.
> > > While
> > > > those flows in MiNiFi are not real-time changes we are dealing
> with
> > > the
> > > > same design elements in creating directed graphs. Additionally,
> > > while
> > > > MiNiFi instances will likely not ever have the same breadth of
> > > extensions
> > > > supported, there is nothing inherently prohibitive of a more
> > complex
> > > graph
> > > > comprised of larger numbers of edges and/or vertices. In my
> view,
> > > the
> > > > mechanics are the same but just being performed in a different
> > > contexts.
> > > > In the case of MiNiFi, this is effectively an “offline” context.
> > By
> > > > "offline," I mean we are not strictly bound to loaded components
> > > such as
> > > > that in the current NiFi context consisting of items from
> unpacked
> > > NARs.
> > > >
> > > > How can we minimize the impact to NiFi?
> > > >
> > > > Given that this is a new capability, I don’t know that we
> > > necessarily need
> > > > to interfere with the existing APIs and could possibly also make
> > > this an
> > > > optional feature. From this standpoint, we could develop new
> > > endpoints for
> > > > this specific context. Some of this work would rely on external
> > > services
> > > > and/or definitions to drive the different context. For example,
> > > without
> > > > NARs acting as the source of available components, something like
> > > the C2
> > > > server could provide extensions for MiNiFi instances it is aware
> of
> > > to help
> > > > drive the experience. This data would be a skeletal view of a
> > > component
> > > > definition/interface without any of the inherent code backing it
> > > needed.
> > > >
> > > > How do we maximize the utility of these efforts?
> > > >
> > > > With the idea about powering MiNiFi flow design with specific
> sets
> > of
> > > > components, I could see this extending to even NiFi itself.
> > > Consider the
> > > > use case where something like I wish to make use of additional
> > > artifacts
> > > > along the lines of [7] and an extension of the current Registry
> > > > implementation. As a simple example, with an understanding of
> what
> > > the
> > > > universe of components is available to my system and not just
> those
> > > > currently loaded, it would be possible for users to design for
> > other
> > > > instances. With these hooks in place, there are a few
> permutations
> > > of this
> > > > relative concept that enable us to carry out some of the “nice to
> > > haves” we
> > > > haven’t quite been able to address.
> > > >
> > > > How does this all come together for MiNiFi?
> > > >
> > > > I have created a work of sheer artistic mastery on the Command
> and
> > > Control
> > > > proposal to illustrate a rough sequence and architecture. [8]
> This
> > > diagram
> > > > shows from entering MiNiFi flow design through to an instance
> > > updating to
> > > > the new flow.
> > > >
> > > >
> > > > Overall, Registry was a tremendous step forward for the NiFi
> > > ecosystem. I
> > > > think in our current state, we have reached a convergence of
> > > capabilities
> > > > that allow us to build higher level abstractions for dataflow
> > > management.
> > > > I think the approach of (2) is a great first foray into that
> realm
> > > that
> > > > provides an immediate solution for a gap in our MiNiFi efforts.
> > > > Additionally, this approach lays the foundation for additional
> > > applications
> > > > possibly inclusive of core NiFi.
> > > >
> > > > I look forward to your thoughts and commentary. Certainly many
> > > details to
> > > > think and reason about, but I see some incredible potential with
> > the
> > > work
> > > > we've done thus far both near and long term.
> > > >
> > > > Thanks for your time!
> > > > --aldrin
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Jeff Zemerick <jz...@apache.org>.
I think it sounds like a good idea. The majority of the MiNiFi flows I have
made are all fairly simple and I have not found the YAML authoring to be
overly burdensome. The more complex flows were done in NiFi and converted.
So as long as the option to make the flows by hand remains (and I don't
think there was anything in the proposal to the contrary), I think it would
be a beneficial addition.
Jeff
On Thu, Mar 1, 2018 at 2:21 PM, Scott Aslan <sc...@gmail.com> wrote:
> AndyC/Aldrin,
>
> The UI/UX design system sub project currently under discussion will be
> based off of the work completed during the NiFi Registry and will not
> contain any UI/UX functionality from NiFi. The long term goal would be to
> upgrade angular in the NiFi UI in order to leverage this new design system.
>
> -Scotty
>
>
> On Thu, Mar 1, 2018 at 1:45 PM, Kevin Doran <kd...@apache.org> wrote:
>
> > Aldrin,
> >
> > Thanks a lot for taking the time to write up such a detailed and well
> > thought out proposal.
> >
> > I really like the idea of publishing flows to NiFi Registry that then get
> > deployed to MiNiFi agents via the C2 Server. I don't have much to add
> > there, and I think that you've identified one of the big missing parts to
> > make that experience user-friendly -- how to design and author the flows
> in
> > the first place.
> >
> > The MiNiFi Flow Designer would fill that need in a really elegant way
> with
> > a low learning curve for NiFi users. Having the ability to design and
> > author flows in an environment other than the "runtime" environment one
> is
> > targeting would be a huge step forward. Not just for MiNiFi, but
> > multi-environment deployments of NiFi, where one might be developing a
> flow
> > in an environment other than where they intend to run it. This would make
> > the SDLC options for NiFi flows even more flexible, which is a big part
> of
> > the goal of the new NiFi Registry.
> >
> > It certainly presents some challenges, but I definitely think the upside
> > for MiNiFi flow authoring would be huge.
> >
> > Thanks,
> > Kevin
> >
> > On 2/28/18, 07:53, "Aldrin Piri" <al...@gmail.com> wrote:
> >
> > It appears I accidentally left off the actual links. Here they are
> in
> > all
> > their glory:
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/MINIFI/
> > MiNiFi+Command+and+Control
> > [2]
> > https://github.com/apache/nifi-minifi/blob/master/
> > minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
> > [3] https://issues.apache.org/jira/browse/MINIFICPP-36
> > [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
> > [5]
> > https://github.com/apache/nifi-minifi/tree/master/
> > minifi-c2#configuration-providers
> > [6]
> > https://github.com/apache/nifi-minifi/blob/master/
> > minifi-docs/src/main/markdown/System_Admin_Guide.md#
> > automatic-warm-redeploy
> > [7]
> > https://cwiki.apache.org/confluence/display/NIFI/
> > Extension+Repositories+%28aka+Extension+Registry%29+for+
> > Dynamically-loaded+Extensions
> > [8]
> > https://cwiki.apache.org/confluence/display/MINIFI/
> > MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
> >
> >
> > On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <al...@gmail.com>
> > wrote:
> >
> > > Hey folks,
> > >
> > > With the release of Registry, I’ve been contemplating leveraging it
> > and
> > > the current codebase of NiFi to facilitate ease of use for MiNiFi
> > flow
> > > design. One of the areas I would like to form a concerted effort
> > around is
> > > that of the Command and Control (C2) functionality originally
> > presented
> > > when the MiNiFi effort was started and further expounded upon with
> a
> > > feature proposal [1]. In that proposal, while the names are dated,
> > we have
> > > components that fulfill some of the core building blocks toward the
> > overall
> > > vision of C2.
> > >
> > > For those not intimately familiar, MiNiFi is primarily configured
> > via a
> > > YAML configuration file. [2] The initial idea was that flows
> would
> > be
> > > simple a source, maybe a transform or two, and then transmission.
> > The
> > > notion was that hand creation of one of these files would not be
> > overly
> > > onerous. It became apparent from questions, however, that users
> > were doing
> > > more complex flows and this YAML ranged from tedious to difficult
> > for those
> > > situations. This precipitated the best approach currently
> available,
> > > utilizing a NiFi instance as a way to generate a template. This
> > template
> > > was then exported, and using our provided toolkit to get YAML out
> > that is
> > > deployable with your instance. Better, but far from ideal.
> > >
> > > A couple approaches came to mind to further narrow this gap of user
> > > expectation and realities of our configuration.
> > >
> > > (1) In our current state, we have heavily relied on convenience of
> > > component overlap. Generally speaking, we relied on the notion
> that
> > NiFi’s
> > > bundled components are a superset of those found in MiNiFi
> > > implementations. We could have continued that, but this would not
> > > necessarily free the user of the file handling and transformation
> > > processes. Additionally, this comes with caveats, among them that
> > given
> > > the ways in which C++ allows us different accesses, there are some
> > > interesting processors that work better for that more native
> > approach and
> > > might not ever justify having an equivalent in Java.
> > >
> > > (2) Providing a first class experience in NiFi for MiNiFi flow
> > designing
> > > making use of the existing UI components as a base and a new design
> > mode
> > > specifically for providing a clean user experience, letting users
> > operate
> > > in a different context but a familiar environment. Certainly
> > preferred,
> > > but how this fit was not clear.
> > >
> > > As a crash course in MiNiFi efforts, there have been some key
> > > contributions that I believe when more strongly united can generate
> > a very
> > > complete authoring and flow update process. Some of these items
> are:
> > >
> > > - C2 API Protocol specification and implementation in C++ [3]
> > > - C2 Server with the notion of serving different flows [4]
> backed
> > by
> > > varying implementations, one of which was via REST to access
> NiFi
> > Templates
> > > directly [5]
> > > - Warm Redeploy in MiNiFi Java [6]
> > >
> > >
> > > One of the first questions to come to mind was, “Is within NiFi the
> > right
> > > place?”
> > >
> > > I think there are several reasons that this is reasonable. Seeing
> > some of
> > > the discussion around the consolidation of common UI components
> > presented
> > > by Scott Aslan, I think sharing the core of the UI elements between
> > the
> > > traditional NiFi and MiNiFi flow designs is a logical extension.
> > While
> > > those flows in MiNiFi are not real-time changes we are dealing with
> > the
> > > same design elements in creating directed graphs. Additionally,
> > while
> > > MiNiFi instances will likely not ever have the same breadth of
> > extensions
> > > supported, there is nothing inherently prohibitive of a more
> complex
> > graph
> > > comprised of larger numbers of edges and/or vertices. In my view,
> > the
> > > mechanics are the same but just being performed in a different
> > contexts.
> > > In the case of MiNiFi, this is effectively an “offline” context.
> By
> > > "offline," I mean we are not strictly bound to loaded components
> > such as
> > > that in the current NiFi context consisting of items from unpacked
> > NARs.
> > >
> > > How can we minimize the impact to NiFi?
> > >
> > > Given that this is a new capability, I don’t know that we
> > necessarily need
> > > to interfere with the existing APIs and could possibly also make
> > this an
> > > optional feature. From this standpoint, we could develop new
> > endpoints for
> > > this specific context. Some of this work would rely on external
> > services
> > > and/or definitions to drive the different context. For example,
> > without
> > > NARs acting as the source of available components, something like
> > the C2
> > > server could provide extensions for MiNiFi instances it is aware of
> > to help
> > > drive the experience. This data would be a skeletal view of a
> > component
> > > definition/interface without any of the inherent code backing it
> > needed.
> > >
> > > How do we maximize the utility of these efforts?
> > >
> > > With the idea about powering MiNiFi flow design with specific sets
> of
> > > components, I could see this extending to even NiFi itself.
> > Consider the
> > > use case where something like I wish to make use of additional
> > artifacts
> > > along the lines of [7] and an extension of the current Registry
> > > implementation. As a simple example, with an understanding of what
> > the
> > > universe of components is available to my system and not just those
> > > currently loaded, it would be possible for users to design for
> other
> > > instances. With these hooks in place, there are a few permutations
> > of this
> > > relative concept that enable us to carry out some of the “nice to
> > haves” we
> > > haven’t quite been able to address.
> > >
> > > How does this all come together for MiNiFi?
> > >
> > > I have created a work of sheer artistic mastery on the Command and
> > Control
> > > proposal to illustrate a rough sequence and architecture. [8] This
> > diagram
> > > shows from entering MiNiFi flow design through to an instance
> > updating to
> > > the new flow.
> > >
> > >
> > > Overall, Registry was a tremendous step forward for the NiFi
> > ecosystem. I
> > > think in our current state, we have reached a convergence of
> > capabilities
> > > that allow us to build higher level abstractions for dataflow
> > management.
> > > I think the approach of (2) is a great first foray into that realm
> > that
> > > provides an immediate solution for a gap in our MiNiFi efforts.
> > > Additionally, this approach lays the foundation for additional
> > applications
> > > possibly inclusive of core NiFi.
> > >
> > > I look forward to your thoughts and commentary. Certainly many
> > details to
> > > think and reason about, but I see some incredible potential with
> the
> > work
> > > we've done thus far both near and long term.
> > >
> > > Thanks for your time!
> > > --aldrin
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Scott Aslan <sc...@gmail.com>.
AndyC/Aldrin,
The UI/UX design system sub project currently under discussion will be
based off of the work completed during the NiFi Registry and will not
contain any UI/UX functionality from NiFi. The long term goal would be to
upgrade angular in the NiFi UI in order to leverage this new design system.
-Scotty
On Thu, Mar 1, 2018 at 1:45 PM, Kevin Doran <kd...@apache.org> wrote:
> Aldrin,
>
> Thanks a lot for taking the time to write up such a detailed and well
> thought out proposal.
>
> I really like the idea of publishing flows to NiFi Registry that then get
> deployed to MiNiFi agents via the C2 Server. I don't have much to add
> there, and I think that you've identified one of the big missing parts to
> make that experience user-friendly -- how to design and author the flows in
> the first place.
>
> The MiNiFi Flow Designer would fill that need in a really elegant way with
> a low learning curve for NiFi users. Having the ability to design and
> author flows in an environment other than the "runtime" environment one is
> targeting would be a huge step forward. Not just for MiNiFi, but
> multi-environment deployments of NiFi, where one might be developing a flow
> in an environment other than where they intend to run it. This would make
> the SDLC options for NiFi flows even more flexible, which is a big part of
> the goal of the new NiFi Registry.
>
> It certainly presents some challenges, but I definitely think the upside
> for MiNiFi flow authoring would be huge.
>
> Thanks,
> Kevin
>
> On 2/28/18, 07:53, "Aldrin Piri" <al...@gmail.com> wrote:
>
> It appears I accidentally left off the actual links. Here they are in
> all
> their glory:
>
> [1]
> https://cwiki.apache.org/confluence/display/MINIFI/
> MiNiFi+Command+and+Control
> [2]
> https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
> [3] https://issues.apache.org/jira/browse/MINIFICPP-36
> [4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
> [5]
> https://github.com/apache/nifi-minifi/tree/master/
> minifi-c2#configuration-providers
> [6]
> https://github.com/apache/nifi-minifi/blob/master/
> minifi-docs/src/main/markdown/System_Admin_Guide.md#
> automatic-warm-redeploy
> [7]
> https://cwiki.apache.org/confluence/display/NIFI/
> Extension+Repositories+%28aka+Extension+Registry%29+for+
> Dynamically-loaded+Extensions
> [8]
> https://cwiki.apache.org/confluence/display/MINIFI/
> MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
>
>
> On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <al...@gmail.com>
> wrote:
>
> > Hey folks,
> >
> > With the release of Registry, I’ve been contemplating leveraging it
> and
> > the current codebase of NiFi to facilitate ease of use for MiNiFi
> flow
> > design. One of the areas I would like to form a concerted effort
> around is
> > that of the Command and Control (C2) functionality originally
> presented
> > when the MiNiFi effort was started and further expounded upon with a
> > feature proposal [1]. In that proposal, while the names are dated,
> we have
> > components that fulfill some of the core building blocks toward the
> overall
> > vision of C2.
> >
> > For those not intimately familiar, MiNiFi is primarily configured
> via a
> > YAML configuration file. [2] The initial idea was that flows would
> be
> > simple a source, maybe a transform or two, and then transmission.
> The
> > notion was that hand creation of one of these files would not be
> overly
> > onerous. It became apparent from questions, however, that users
> were doing
> > more complex flows and this YAML ranged from tedious to difficult
> for those
> > situations. This precipitated the best approach currently available,
> > utilizing a NiFi instance as a way to generate a template. This
> template
> > was then exported, and using our provided toolkit to get YAML out
> that is
> > deployable with your instance. Better, but far from ideal.
> >
> > A couple approaches came to mind to further narrow this gap of user
> > expectation and realities of our configuration.
> >
> > (1) In our current state, we have heavily relied on convenience of
> > component overlap. Generally speaking, we relied on the notion that
> NiFi’s
> > bundled components are a superset of those found in MiNiFi
> > implementations. We could have continued that, but this would not
> > necessarily free the user of the file handling and transformation
> > processes. Additionally, this comes with caveats, among them that
> given
> > the ways in which C++ allows us different accesses, there are some
> > interesting processors that work better for that more native
> approach and
> > might not ever justify having an equivalent in Java.
> >
> > (2) Providing a first class experience in NiFi for MiNiFi flow
> designing
> > making use of the existing UI components as a base and a new design
> mode
> > specifically for providing a clean user experience, letting users
> operate
> > in a different context but a familiar environment. Certainly
> preferred,
> > but how this fit was not clear.
> >
> > As a crash course in MiNiFi efforts, there have been some key
> > contributions that I believe when more strongly united can generate
> a very
> > complete authoring and flow update process. Some of these items are:
> >
> > - C2 API Protocol specification and implementation in C++ [3]
> > - C2 Server with the notion of serving different flows [4] backed
> by
> > varying implementations, one of which was via REST to access NiFi
> Templates
> > directly [5]
> > - Warm Redeploy in MiNiFi Java [6]
> >
> >
> > One of the first questions to come to mind was, “Is within NiFi the
> right
> > place?”
> >
> > I think there are several reasons that this is reasonable. Seeing
> some of
> > the discussion around the consolidation of common UI components
> presented
> > by Scott Aslan, I think sharing the core of the UI elements between
> the
> > traditional NiFi and MiNiFi flow designs is a logical extension.
> While
> > those flows in MiNiFi are not real-time changes we are dealing with
> the
> > same design elements in creating directed graphs. Additionally,
> while
> > MiNiFi instances will likely not ever have the same breadth of
> extensions
> > supported, there is nothing inherently prohibitive of a more complex
> graph
> > comprised of larger numbers of edges and/or vertices. In my view,
> the
> > mechanics are the same but just being performed in a different
> contexts.
> > In the case of MiNiFi, this is effectively an “offline” context. By
> > "offline," I mean we are not strictly bound to loaded components
> such as
> > that in the current NiFi context consisting of items from unpacked
> NARs.
> >
> > How can we minimize the impact to NiFi?
> >
> > Given that this is a new capability, I don’t know that we
> necessarily need
> > to interfere with the existing APIs and could possibly also make
> this an
> > optional feature. From this standpoint, we could develop new
> endpoints for
> > this specific context. Some of this work would rely on external
> services
> > and/or definitions to drive the different context. For example,
> without
> > NARs acting as the source of available components, something like
> the C2
> > server could provide extensions for MiNiFi instances it is aware of
> to help
> > drive the experience. This data would be a skeletal view of a
> component
> > definition/interface without any of the inherent code backing it
> needed.
> >
> > How do we maximize the utility of these efforts?
> >
> > With the idea about powering MiNiFi flow design with specific sets of
> > components, I could see this extending to even NiFi itself.
> Consider the
> > use case where something like I wish to make use of additional
> artifacts
> > along the lines of [7] and an extension of the current Registry
> > implementation. As a simple example, with an understanding of what
> the
> > universe of components is available to my system and not just those
> > currently loaded, it would be possible for users to design for other
> > instances. With these hooks in place, there are a few permutations
> of this
> > relative concept that enable us to carry out some of the “nice to
> haves” we
> > haven’t quite been able to address.
> >
> > How does this all come together for MiNiFi?
> >
> > I have created a work of sheer artistic mastery on the Command and
> Control
> > proposal to illustrate a rough sequence and architecture. [8] This
> diagram
> > shows from entering MiNiFi flow design through to an instance
> updating to
> > the new flow.
> >
> >
> > Overall, Registry was a tremendous step forward for the NiFi
> ecosystem. I
> > think in our current state, we have reached a convergence of
> capabilities
> > that allow us to build higher level abstractions for dataflow
> management.
> > I think the approach of (2) is a great first foray into that realm
> that
> > provides an immediate solution for a gap in our MiNiFi efforts.
> > Additionally, this approach lays the foundation for additional
> applications
> > possibly inclusive of core NiFi.
> >
> > I look forward to your thoughts and commentary. Certainly many
> details to
> > think and reason about, but I see some incredible potential with the
> work
> > we've done thus far both near and long term.
> >
> > Thanks for your time!
> > --aldrin
> >
> >
>
>
>
>
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Kevin Doran <kd...@apache.org>.
Aldrin,
Thanks a lot for taking the time to write up such a detailed and well thought out proposal.
I really like the idea of publishing flows to NiFi Registry that then get deployed to MiNiFi agents via the C2 Server. I don't have much to add there, and I think that you've identified one of the big missing parts to make that experience user-friendly -- how to design and author the flows in the first place.
The MiNiFi Flow Designer would fill that need in a really elegant way with a low learning curve for NiFi users. Having the ability to design and author flows in an environment other than the "runtime" environment one is targeting would be a huge step forward. Not just for MiNiFi, but multi-environment deployments of NiFi, where one might be developing a flow in an environment other than where they intend to run it. This would make the SDLC options for NiFi flows even more flexible, which is a big part of the goal of the new NiFi Registry.
It certainly presents some challenges, but I definitely think the upside for MiNiFi flow authoring would be huge.
Thanks,
Kevin
On 2/28/18, 07:53, "Aldrin Piri" <al...@gmail.com> wrote:
It appears I accidentally left off the actual links. Here they are in all
their glory:
[1]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
[2]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
[3] https://issues.apache.org/jira/browse/MINIFICPP-36
[4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
[5]
https://github.com/apache/nifi-minifi/tree/master/minifi-c2#configuration-providers
[6]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#automatic-warm-redeploy
[7]
https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
[8]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <al...@gmail.com> wrote:
> Hey folks,
>
> With the release of Registry, I’ve been contemplating leveraging it and
> the current codebase of NiFi to facilitate ease of use for MiNiFi flow
> design. One of the areas I would like to form a concerted effort around is
> that of the Command and Control (C2) functionality originally presented
> when the MiNiFi effort was started and further expounded upon with a
> feature proposal [1]. In that proposal, while the names are dated, we have
> components that fulfill some of the core building blocks toward the overall
> vision of C2.
>
> For those not intimately familiar, MiNiFi is primarily configured via a
> YAML configuration file. [2] The initial idea was that flows would be
> simple a source, maybe a transform or two, and then transmission. The
> notion was that hand creation of one of these files would not be overly
> onerous. It became apparent from questions, however, that users were doing
> more complex flows and this YAML ranged from tedious to difficult for those
> situations. This precipitated the best approach currently available,
> utilizing a NiFi instance as a way to generate a template. This template
> was then exported, and using our provided toolkit to get YAML out that is
> deployable with your instance. Better, but far from ideal.
>
> A couple approaches came to mind to further narrow this gap of user
> expectation and realities of our configuration.
>
> (1) In our current state, we have heavily relied on convenience of
> component overlap. Generally speaking, we relied on the notion that NiFi’s
> bundled components are a superset of those found in MiNiFi
> implementations. We could have continued that, but this would not
> necessarily free the user of the file handling and transformation
> processes. Additionally, this comes with caveats, among them that given
> the ways in which C++ allows us different accesses, there are some
> interesting processors that work better for that more native approach and
> might not ever justify having an equivalent in Java.
>
> (2) Providing a first class experience in NiFi for MiNiFi flow designing
> making use of the existing UI components as a base and a new design mode
> specifically for providing a clean user experience, letting users operate
> in a different context but a familiar environment. Certainly preferred,
> but how this fit was not clear.
>
> As a crash course in MiNiFi efforts, there have been some key
> contributions that I believe when more strongly united can generate a very
> complete authoring and flow update process. Some of these items are:
>
> - C2 API Protocol specification and implementation in C++ [3]
> - C2 Server with the notion of serving different flows [4] backed by
> varying implementations, one of which was via REST to access NiFi Templates
> directly [5]
> - Warm Redeploy in MiNiFi Java [6]
>
>
> One of the first questions to come to mind was, “Is within NiFi the right
> place?”
>
> I think there are several reasons that this is reasonable. Seeing some of
> the discussion around the consolidation of common UI components presented
> by Scott Aslan, I think sharing the core of the UI elements between the
> traditional NiFi and MiNiFi flow designs is a logical extension. While
> those flows in MiNiFi are not real-time changes we are dealing with the
> same design elements in creating directed graphs. Additionally, while
> MiNiFi instances will likely not ever have the same breadth of extensions
> supported, there is nothing inherently prohibitive of a more complex graph
> comprised of larger numbers of edges and/or vertices. In my view, the
> mechanics are the same but just being performed in a different contexts.
> In the case of MiNiFi, this is effectively an “offline” context. By
> "offline," I mean we are not strictly bound to loaded components such as
> that in the current NiFi context consisting of items from unpacked NARs.
>
> How can we minimize the impact to NiFi?
>
> Given that this is a new capability, I don’t know that we necessarily need
> to interfere with the existing APIs and could possibly also make this an
> optional feature. From this standpoint, we could develop new endpoints for
> this specific context. Some of this work would rely on external services
> and/or definitions to drive the different context. For example, without
> NARs acting as the source of available components, something like the C2
> server could provide extensions for MiNiFi instances it is aware of to help
> drive the experience. This data would be a skeletal view of a component
> definition/interface without any of the inherent code backing it needed.
>
> How do we maximize the utility of these efforts?
>
> With the idea about powering MiNiFi flow design with specific sets of
> components, I could see this extending to even NiFi itself. Consider the
> use case where something like I wish to make use of additional artifacts
> along the lines of [7] and an extension of the current Registry
> implementation. As a simple example, with an understanding of what the
> universe of components is available to my system and not just those
> currently loaded, it would be possible for users to design for other
> instances. With these hooks in place, there are a few permutations of this
> relative concept that enable us to carry out some of the “nice to haves” we
> haven’t quite been able to address.
>
> How does this all come together for MiNiFi?
>
> I have created a work of sheer artistic mastery on the Command and Control
> proposal to illustrate a rough sequence and architecture. [8] This diagram
> shows from entering MiNiFi flow design through to an instance updating to
> the new flow.
>
>
> Overall, Registry was a tremendous step forward for the NiFi ecosystem. I
> think in our current state, we have reached a convergence of capabilities
> that allow us to build higher level abstractions for dataflow management.
> I think the approach of (2) is a great first foray into that realm that
> provides an immediate solution for a gap in our MiNiFi efforts.
> Additionally, this approach lays the foundation for additional applications
> possibly inclusive of core NiFi.
>
> I look forward to your thoughts and commentary. Certainly many details to
> think and reason about, but I see some incredible potential with the work
> we've done thus far both near and long term.
>
> Thanks for your time!
> --aldrin
>
>
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Aldrin Piri <al...@gmail.com>.
It appears I accidentally left off the actual links. Here they are in all
their glory:
[1]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
[2]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#config-file
[3] https://issues.apache.org/jira/browse/MINIFICPP-36
[4] https://github.com/apache/nifi-minifi/tree/master/minifi-c2
[5]
https://github.com/apache/nifi-minifi/tree/master/minifi-c2#configuration-providers
[6]
https://github.com/apache/nifi-minifi/blob/master/minifi-docs/src/main/markdown/System_Admin_Guide.md#automatic-warm-redeploy
[7]
https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
[8]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control#MiNiFiCommandandControl-FlowAuthorshipDetails
On Wed, Feb 28, 2018 at 1:53 AM, Aldrin Piri <al...@gmail.com> wrote:
> Hey folks,
>
> With the release of Registry, I’ve been contemplating leveraging it and
> the current codebase of NiFi to facilitate ease of use for MiNiFi flow
> design. One of the areas I would like to form a concerted effort around is
> that of the Command and Control (C2) functionality originally presented
> when the MiNiFi effort was started and further expounded upon with a
> feature proposal [1]. In that proposal, while the names are dated, we have
> components that fulfill some of the core building blocks toward the overall
> vision of C2.
>
> For those not intimately familiar, MiNiFi is primarily configured via a
> YAML configuration file. [2] The initial idea was that flows would be
> simple a source, maybe a transform or two, and then transmission. The
> notion was that hand creation of one of these files would not be overly
> onerous. It became apparent from questions, however, that users were doing
> more complex flows and this YAML ranged from tedious to difficult for those
> situations. This precipitated the best approach currently available,
> utilizing a NiFi instance as a way to generate a template. This template
> was then exported, and using our provided toolkit to get YAML out that is
> deployable with your instance. Better, but far from ideal.
>
> A couple approaches came to mind to further narrow this gap of user
> expectation and realities of our configuration.
>
> (1) In our current state, we have heavily relied on convenience of
> component overlap. Generally speaking, we relied on the notion that NiFi’s
> bundled components are a superset of those found in MiNiFi
> implementations. We could have continued that, but this would not
> necessarily free the user of the file handling and transformation
> processes. Additionally, this comes with caveats, among them that given
> the ways in which C++ allows us different accesses, there are some
> interesting processors that work better for that more native approach and
> might not ever justify having an equivalent in Java.
>
> (2) Providing a first class experience in NiFi for MiNiFi flow designing
> making use of the existing UI components as a base and a new design mode
> specifically for providing a clean user experience, letting users operate
> in a different context but a familiar environment. Certainly preferred,
> but how this fit was not clear.
>
> As a crash course in MiNiFi efforts, there have been some key
> contributions that I believe when more strongly united can generate a very
> complete authoring and flow update process. Some of these items are:
>
> - C2 API Protocol specification and implementation in C++ [3]
> - C2 Server with the notion of serving different flows [4] backed by
> varying implementations, one of which was via REST to access NiFi Templates
> directly [5]
> - Warm Redeploy in MiNiFi Java [6]
>
>
> One of the first questions to come to mind was, “Is within NiFi the right
> place?”
>
> I think there are several reasons that this is reasonable. Seeing some of
> the discussion around the consolidation of common UI components presented
> by Scott Aslan, I think sharing the core of the UI elements between the
> traditional NiFi and MiNiFi flow designs is a logical extension. While
> those flows in MiNiFi are not real-time changes we are dealing with the
> same design elements in creating directed graphs. Additionally, while
> MiNiFi instances will likely not ever have the same breadth of extensions
> supported, there is nothing inherently prohibitive of a more complex graph
> comprised of larger numbers of edges and/or vertices. In my view, the
> mechanics are the same but just being performed in a different contexts.
> In the case of MiNiFi, this is effectively an “offline” context. By
> "offline," I mean we are not strictly bound to loaded components such as
> that in the current NiFi context consisting of items from unpacked NARs.
>
> How can we minimize the impact to NiFi?
>
> Given that this is a new capability, I don’t know that we necessarily need
> to interfere with the existing APIs and could possibly also make this an
> optional feature. From this standpoint, we could develop new endpoints for
> this specific context. Some of this work would rely on external services
> and/or definitions to drive the different context. For example, without
> NARs acting as the source of available components, something like the C2
> server could provide extensions for MiNiFi instances it is aware of to help
> drive the experience. This data would be a skeletal view of a component
> definition/interface without any of the inherent code backing it needed.
>
> How do we maximize the utility of these efforts?
>
> With the idea about powering MiNiFi flow design with specific sets of
> components, I could see this extending to even NiFi itself. Consider the
> use case where something like I wish to make use of additional artifacts
> along the lines of [7] and an extension of the current Registry
> implementation. As a simple example, with an understanding of what the
> universe of components is available to my system and not just those
> currently loaded, it would be possible for users to design for other
> instances. With these hooks in place, there are a few permutations of this
> relative concept that enable us to carry out some of the “nice to haves” we
> haven’t quite been able to address.
>
> How does this all come together for MiNiFi?
>
> I have created a work of sheer artistic mastery on the Command and Control
> proposal to illustrate a rough sequence and architecture. [8] This diagram
> shows from entering MiNiFi flow design through to an instance updating to
> the new flow.
>
>
> Overall, Registry was a tremendous step forward for the NiFi ecosystem. I
> think in our current state, we have reached a convergence of capabilities
> that allow us to build higher level abstractions for dataflow management.
> I think the approach of (2) is a great first foray into that realm that
> provides an immediate solution for a gap in our MiNiFi efforts.
> Additionally, this approach lays the foundation for additional applications
> possibly inclusive of core NiFi.
>
> I look forward to your thoughts and commentary. Certainly many details to
> think and reason about, but I see some incredible potential with the work
> we've done thus far both near and long term.
>
> Thanks for your time!
> --aldrin
>
>
Re: [DISCUSS] MiNiFi Flow Designer
Posted by Andy Christianson <ai...@protonmail.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
> I think there are several reasons that this is reasonable. Seeing some of
> the discussion around the consolidation of common UI components presented
> by Scott Aslan, I think sharing the core of the UI elements between the
> traditional NiFi and MiNiFi flow designs is a logical extension. While
> those flows in MiNiFi are not real-time changes we are dealing with the
> same design elements in creating directed graphs. Additionally, while
> MiNiFi instances will likely not ever have the same breadth of extensions
> supported, there is nothing inherently prohibitive of a more complex graph
> comprised of larger numbers of edges and/or vertices. In my view, the
> mechanics are the same but just being performed in a different contexts.
> In the case of MiNiFi, this is effectively an “offline” context. By
> "offline," I mean we are not strictly bound to loaded components such as
> that in the current NiFi context consisting of items from unpacked NARs.
A unified UI, or at least common code base/components makes a lot of sense.
MiNiFi flows are essentially NiFi flows and so it makes sense to design them
with the same UI.
That said, recent improvements have made it less burdonsome to hand-write the
YAML, and leaving that option makes sense to accomodate users who do not wish
to use a graphical flow designer. I think it's worth exploring further
improvements to the textual configuration landscape.
> How can we minimize the impact to NiFi?
With the emergence of multiple NiFi "execution engines," such as MiNiFi - C++
and MiNiFi - Java, it almost seems to make sense to decouple the UI from NiFi,
leaving the execution engine and UI as separate pieces. The decoupled Flow
Designer would then be the sole, unified tool for graphically designing flows.
This also enables the possibility of using other tools, such as declarative
flow definition files (something like Graphviz dot files, or an improved
version of our YAML syntax) to configure NiFi. This might be a boon to the
product in certain deployment environments/scenarios.
Regards,
Andy I.C.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQEcBAEBAgAGBQJamCr2AAoJEG1+mBKNMpIDqs8H/3OAHIs3QDoQo+dmnhrZTlRY
jCzUBOM2MM9ogs265p3WGNtpSeEH94o6+nCn8eAYpW1g1NZDd93JEvTcUjFPx1Vs
jlgmyOdmqoOZvRb5z4Y2vSjRf76D8QwNE8NMDXQtSxhvshRgOuYpaDxFmoC5dlRI
5XlRKXoHucvpIel2gBS7vEZdeWtc1PBNaxTwbWu+ESjhLEmSD+gtYtTZ/B6oCifI
LkjOCNxRqoA9Du0GDUO7KlOL4ZzcYUI7xNyJFTvzaugIxlX1nAXlgplTz24Pffsc
N2+DLvM0RKmjMINfOEdnArHHA0avQvH8NGj1WCoKUcPWmIKthgTWnDJXtaYLKrk=
=Kagd
-----END PGP SIGNATURE-----
Sent from ProtonMail, Swiss-based encrypted email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On February 28, 2018 1:53 AM, Aldrin Piri <al...@gmail.com> wrote:
>
>
> Hey folks,
>
> With the release of Registry, I’ve been contemplating leveraging it and the
>
> current codebase of NiFi to facilitate ease of use for MiNiFi flow design.
>
> One of the areas I would like to form a concerted effort around is that of
>
> the Command and Control (C2) functionality originally presented when the
>
> MiNiFi effort was started and further expounded upon with a feature
>
> proposal \[1\]. In that proposal, while the names are dated, we have
>
> components that fulfill some of the core building blocks toward the overall
>
> vision of C2.
>
> For those not intimately familiar, MiNiFi is primarily configured via a
>
> YAML configuration file. \[2\] The initial idea was that flows would be
>
> simple a source, maybe a transform or two, and then transmission. The
>
> notion was that hand creation of one of these files would not be overly
>
> onerous. It became apparent from questions, however, that users were doing
>
> more complex flows and this YAML ranged from tedious to difficult for those
>
> situations. This precipitated the best approach currently available,
>
> utilizing a NiFi instance as a way to generate a template. This template
>
> was then exported, and using our provided toolkit to get YAML out that is
>
> deployable with your instance. Better, but far from ideal.
>
> A couple approaches came to mind to further narrow this gap of user
>
> expectation and realities of our configuration.
>
> (1) In our current state, we have heavily relied on convenience of
>
> component overlap. Generally speaking, we relied on the notion that NiFi’s
>
> bundled components are a superset of those found in MiNiFi
>
> implementations. We could have continued that, but this would not
>
> necessarily free the user of the file handling and transformation
>
> processes. Additionally, this comes with caveats, among them that given
>
> the ways in which C++ allows us different accesses, there are some
>
> interesting processors that work better for that more native approach and
>
> might not ever justify having an equivalent in Java.
>
> (2) Providing a first class experience in NiFi for MiNiFi flow designing
>
> making use of the existing UI components as a base and a new design mode
>
> specifically for providing a clean user experience, letting users operate
>
> in a different context but a familiar environment. Certainly preferred,
>
> but how this fit was not clear.
>
> As a crash course in MiNiFi efforts, there have been some key contributions
>
> that I believe when more strongly united can generate a very complete
>
> authoring and flow update process. Some of these items are:
>
> - C2 API Protocol specification and implementation in C++ \[3\]
> - C2 Server with the notion of serving different flows \[4\] backed by
>
> varying implementations, one of which was via REST to access NiFi Templates
>
> directly \[5\]
>
> - Warm Redeploy in MiNiFi Java \[6\]
>
> One of the first questions to come to mind was, “Is within NiFi the right
>
> place?”
>
> I think there are several reasons that this is reasonable. Seeing some of
>
> the discussion around the consolidation of common UI components presented
>
> by Scott Aslan, I think sharing the core of the UI elements between the
>
> traditional NiFi and MiNiFi flow designs is a logical extension. While
>
> those flows in MiNiFi are not real-time changes we are dealing with the
>
> same design elements in creating directed graphs. Additionally, while
>
> MiNiFi instances will likely not ever have the same breadth of extensions
>
> supported, there is nothing inherently prohibitive of a more complex graph
>
> comprised of larger numbers of edges and/or vertices. In my view, the
>
> mechanics are the same but just being performed in a different contexts.
>
> In the case of MiNiFi, this is effectively an “offline” context. By
>
> "offline," I mean we are not strictly bound to loaded components such as
>
> that in the current NiFi context consisting of items from unpacked NARs.
>
> How can we minimize the impact to NiFi?
>
> Given that this is a new capability, I don’t know that we necessarily need
>
> to interfere with the existing APIs and could possibly also make this an
>
> optional feature. From this standpoint, we could develop new endpoints for
>
> this specific context. Some of this work would rely on external services
>
> and/or definitions to drive the different context. For example, without
>
> NARs acting as the source of available components, something like the C2
>
> server could provide extensions for MiNiFi instances it is aware of to help
>
> drive the experience. This data would be a skeletal view of a component
>
> definition/interface without any of the inherent code backing it needed.
>
> How do we maximize the utility of these efforts?
>
> With the idea about powering MiNiFi flow design with specific sets of
>
> components, I could see this extending to even NiFi itself. Consider the
>
> use case where something like I wish to make use of additional artifacts
>
> along the lines of \[7\] and an extension of the current Registry
>
> implementation. As a simple example, with an understanding of what the
>
> universe of components is available to my system and not just those
>
> currently loaded, it would be possible for users to design for other
>
> instances. With these hooks in place, there are a few permutations of this
>
> relative concept that enable us to carry out some of the “nice to haves” we
>
> haven’t quite been able to address.
>
> How does this all come together for MiNiFi?
>
> I have created a work of sheer artistic mastery on the Command and Control
>
> proposal to illustrate a rough sequence and architecture. \[8\] This diagram
>
> shows from entering MiNiFi flow design through to an instance updating to
>
> the new flow.
>
> Overall, Registry was a tremendous step forward for the NiFi ecosystem. I
>
> think in our current state, we have reached a convergence of capabilities
>
> that allow us to build higher level abstractions for dataflow management.
>
> I think the approach of (2) is a great first foray into that realm that
>
> provides an immediate solution for a gap in our MiNiFi efforts.
>
> Additionally, this approach lays the foundation for additional applications
>
> possibly inclusive of core NiFi.
>
> I look forward to your thoughts and commentary. Certainly many details to
>
> think and reason about, but I see some incredible potential with the work
>
> we've done thus far both near and long term.
>
> Thanks for your time!
>
> --aldrin
>