Re: [DISCUSS] MiNiFi Flow Designer

[Andy Christianson <ai...@protonmail.com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> I think there are several reasons that this is reasonable. Seeing some of
> the discussion around the consolidation of common UI components presented
> by Scott Aslan, I think sharing the core of the UI elements between the
> traditional NiFi and MiNiFi flow designs is a logical extension.  While
> those flows in MiNiFi are not real-time changes we are dealing with the
> same design elements in creating directed graphs.  Additionally, while
> MiNiFi instances will likely not ever have the same breadth of extensions
> supported, there is nothing inherently prohibitive of a more complex graph
> comprised of larger numbers of edges and/or vertices.  In my view, the
> mechanics are the same but just being performed in a different contexts.
> In the case of MiNiFi, this is effectively an “offline” context.  By
> "offline," I mean we are not strictly bound to loaded components such as
> that in the current NiFi context consisting of items from unpacked NARs.

A unified UI, or at least common code base/components makes a lot of sense.
MiNiFi flows are essentially NiFi flows and so it makes sense to design them
with the same UI.

That said, recent improvements have made it less burdonsome to hand-write the
YAML, and leaving that option makes sense to accomodate users who do not wish
to use a graphical flow designer. I think it's worth exploring further
improvements to the textual configuration landscape.

> How can we minimize the impact to NiFi?

With the emergence of multiple NiFi "execution engines," such as MiNiFi - C++
and MiNiFi - Java, it almost seems to make sense to decouple the UI from NiFi,
leaving the execution engine and UI as separate pieces. The decoupled Flow
Designer would then be the sole, unified tool for graphically designing flows.
This also enables the possibility of using other tools, such as declarative
flow definition files (something like Graphviz dot files, or an improved
version of our YAML syntax) to configure NiFi. This might be a boon to the
product in certain deployment environments/scenarios.

Regards,

Andy I.C.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQEcBAEBAgAGBQJamCr2AAoJEG1+mBKNMpIDqs8H/3OAHIs3QDoQo+dmnhrZTlRY
jCzUBOM2MM9ogs265p3WGNtpSeEH94o6+nCn8eAYpW1g1NZDd93JEvTcUjFPx1Vs
jlgmyOdmqoOZvRb5z4Y2vSjRf76D8QwNE8NMDXQtSxhvshRgOuYpaDxFmoC5dlRI
5XlRKXoHucvpIel2gBS7vEZdeWtc1PBNaxTwbWu+ESjhLEmSD+gtYtTZ/B6oCifI
LkjOCNxRqoA9Du0GDUO7KlOL4ZzcYUI7xNyJFTvzaugIxlX1nAXlgplTz24Pffsc
N2+DLvM0RKmjMINfOEdnArHHA0avQvH8NGj1WCoKUcPWmIKthgTWnDJXtaYLKrk=
=Kagd
-----END PGP SIGNATURE-----


​Sent from ProtonMail, Swiss-based encrypted email.​

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On February 28, 2018 1:53 AM, Aldrin Piri <al...@gmail.com> wrote:

> ​​
> 
> Hey folks,
> 
> With the release of Registry, I’ve been contemplating leveraging it and the
> 
> current codebase of NiFi to facilitate ease of use for MiNiFi flow design.
> 
> One of the areas I would like to form a concerted effort around is that of
> 
> the Command and Control (C2) functionality originally presented when the
> 
> MiNiFi effort was started and further expounded upon with a feature
> 
> proposal \[1\]. In that proposal, while the names are dated, we have
> 
> components that fulfill some of the core building blocks toward the overall
> 
> vision of C2.
> 
> For those not intimately familiar, MiNiFi is primarily configured via a
> 
> YAML configuration file. \[2\] The initial idea was that flows would be
> 
> simple a source, maybe a transform or two, and then transmission. The
> 
> notion was that hand creation of one of these files would not be overly
> 
> onerous. It became apparent from questions, however, that users were doing
> 
> more complex flows and this YAML ranged from tedious to difficult for those
> 
> situations. This precipitated the best approach currently available,
> 
> utilizing a NiFi instance as a way to generate a template. This template
> 
> was then exported, and using our provided toolkit to get YAML out that is
> 
> deployable with your instance. Better, but far from ideal.
> 
> A couple approaches came to mind to further narrow this gap of user
> 
> expectation and realities of our configuration.
> 
> (1) In our current state, we have heavily relied on convenience of
> 
> component overlap. Generally speaking, we relied on the notion that NiFi’s
> 
> bundled components are a superset of those found in MiNiFi
> 
> implementations. We could have continued that, but this would not
> 
> necessarily free the user of the file handling and transformation
> 
> processes. Additionally, this comes with caveats, among them that given
> 
> the ways in which C++ allows us different accesses, there are some
> 
> interesting processors that work better for that more native approach and
> 
> might not ever justify having an equivalent in Java.
> 
> (2) Providing a first class experience in NiFi for MiNiFi flow designing
> 
> making use of the existing UI components as a base and a new design mode
> 
> specifically for providing a clean user experience, letting users operate
> 
> in a different context but a familiar environment. Certainly preferred,
> 
> but how this fit was not clear.
> 
> As a crash course in MiNiFi efforts, there have been some key contributions
> 
> that I believe when more strongly united can generate a very complete
> 
> authoring and flow update process. Some of these items are:
> 
> -   C2 API Protocol specification and implementation in C++ \[3\]
> -   C2 Server with the notion of serving different flows \[4\] backed by
>     
>     varying implementations, one of which was via REST to access NiFi Templates
>     
>     directly \[5\]
>     
> -   Warm Redeploy in MiNiFi Java \[6\]
>     
>     One of the first questions to come to mind was, “Is within NiFi the right
>     
>     place?”
>     
>     I think there are several reasons that this is reasonable. Seeing some of
>     
>     the discussion around the consolidation of common UI components presented
>     
>     by Scott Aslan, I think sharing the core of the UI elements between the
>     
>     traditional NiFi and MiNiFi flow designs is a logical extension. While
>     
>     those flows in MiNiFi are not real-time changes we are dealing with the
>     
>     same design elements in creating directed graphs. Additionally, while
>     
>     MiNiFi instances will likely not ever have the same breadth of extensions
>     
>     supported, there is nothing inherently prohibitive of a more complex graph
>     
>     comprised of larger numbers of edges and/or vertices. In my view, the
>     
>     mechanics are the same but just being performed in a different contexts.
>     
>     In the case of MiNiFi, this is effectively an “offline” context. By
>     
>     "offline," I mean we are not strictly bound to loaded components such as
>     
>     that in the current NiFi context consisting of items from unpacked NARs.
>     
>     How can we minimize the impact to NiFi?
>     
>     Given that this is a new capability, I don’t know that we necessarily need
>     
>     to interfere with the existing APIs and could possibly also make this an
>     
>     optional feature. From this standpoint, we could develop new endpoints for
>     
>     this specific context. Some of this work would rely on external services
>     
>     and/or definitions to drive the different context. For example, without
>     
>     NARs acting as the source of available components, something like the C2
>     
>     server could provide extensions for MiNiFi instances it is aware of to help
>     
>     drive the experience. This data would be a skeletal view of a component
>     
>     definition/interface without any of the inherent code backing it needed.
>     
>     How do we maximize the utility of these efforts?
>     
>     With the idea about powering MiNiFi flow design with specific sets of
>     
>     components, I could see this extending to even NiFi itself. Consider the
>     
>     use case where something like I wish to make use of additional artifacts
>     
>     along the lines of \[7\] and an extension of the current Registry
>     
>     implementation. As a simple example, with an understanding of what the
>     
>     universe of components is available to my system and not just those
>     
>     currently loaded, it would be possible for users to design for other
>     
>     instances. With these hooks in place, there are a few permutations of this
>     
>     relative concept that enable us to carry out some of the “nice to haves” we
>     
>     haven’t quite been able to address.
>     
>     How does this all come together for MiNiFi?
>     
>     I have created a work of sheer artistic mastery on the Command and Control
>     
>     proposal to illustrate a rough sequence and architecture. \[8\] This diagram
>     
>     shows from entering MiNiFi flow design through to an instance updating to
>     
>     the new flow.
>     
>     Overall, Registry was a tremendous step forward for the NiFi ecosystem. I
>     
>     think in our current state, we have reached a convergence of capabilities
>     
>     that allow us to build higher level abstractions for dataflow management.
>     
>     I think the approach of (2) is a great first foray into that realm that
>     
>     provides an immediate solution for a gap in our MiNiFi efforts.
>     
>     Additionally, this approach lays the foundation for additional applications
>     
>     possibly inclusive of core NiFi.
>     
>     I look forward to your thoughts and commentary. Certainly many details to
>     
>     think and reason about, but I see some incredible potential with the work
>     
>     we've done thus far both near and long term.
>     
>     Thanks for your time!
>     
>     --aldrin
>