File-based Versioned Flow Representation

[Samuel Hjelmfelt <sa...@yahoo.com.INVALID>]

Hello everyone,NIFI-6539 brings up the topic of deployment scenarios when a NiFi registry is not available. MiNiFi allows configuration via a YML file-based representation of a flow, often created from an exported NiFi template. However, there has been a push to focus on versioned flows in the registry and to move away from templates.
Long term, what is the plan for file-based flow representations? Is the intent to continue using templates even in NiFi 2.x, or is there some other solution planned? Both deployments without a registry and registry import/export will require some kind of file-based flow representation.
-Sam

Re: File-based Versioned Flow Representation

[Andy LoPresto <al...@apache.org>]

I will throw a vote for JSON-based representation moving forward as well, with the understanding that we now have three different codebases which convert from a serialized, human-readable format (XML, YAML, JSON) into the hydrated flow state object and back, and we should ensure through unit, integration, and regression tests that all three function identically (or at least with expected and documented differences). In addition, we should organize those codebases intentionally so that changes to one are applied to the others if relevant, and perhaps organize and expose them in such a way that easy tooling is available around them to convert the various serialized forms to another. 


Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Sep 3, 2019, at 8:16 AM, Kevin Doran <kd...@apache.org> wrote:
> 
> I agree I think that the JSON-serialized version flow snapshot in
> Registry, which is also supported by CLI and MiNiFi Toolkit, should
> become the new "de facto" for portable/file flow format, including
> used in places that templates used to be used.
> 
> +1 for adding import/export capabilities for this format in NiFi
> similar to the way templates work today. It would be nice if the "Add
> Process Group..." dialog had additional options other than import from
> NiFi Registry, such as import from file or even just a text box the
> JSON representation could be pasted into.
> 
> On Fri, Aug 30, 2019 at 9:05 PM Bryan Bende <bb...@gmail.com> wrote:
>> 
>> I’d like to see us move towards the versioned flow representation from
>> registry and add features to NiFi like right click on PG and export to
>> versioned flow, import from a file etc.
>> 
>> The Minifi java toolkit already has a transform from versioned flow JSON to
>> YAML, just like the template transform.
>> 
>> 
>> On Fri, Aug 30, 2019 at 4:05 PM Samuel Hjelmfelt
>> <sa...@yahoo.com.invalid> wrote:
>> 
>>> Hello everyone,NIFI-6539 brings up the topic of deployment scenarios when
>>> a NiFi registry is not available. MiNiFi allows configuration via a YML
>>> file-based representation of a flow, often created from an exported NiFi
>>> template. However, there has been a push to focus on versioned flows in the
>>> registry and to move away from templates.
>>> Long term, what is the plan for file-based flow representations? Is the
>>> intent to continue using templates even in NiFi 2.x, or is there some other
>>> solution planned? Both deployments without a registry and registry
>>> import/export will require some kind of file-based flow representation.
>>> -Sam
>>> 
>> --
>> Sent from Gmail Mobile

Re: File-based Versioned Flow Representation

[Kevin Doran <kd...@apache.org>]

I agree I think that the JSON-serialized version flow snapshot in
Registry, which is also supported by CLI and MiNiFi Toolkit, should
become the new "de facto" for portable/file flow format, including
used in places that templates used to be used.

+1 for adding import/export capabilities for this format in NiFi
similar to the way templates work today. It would be nice if the "Add
Process Group..." dialog had additional options other than import from
NiFi Registry, such as import from file or even just a text box the
JSON representation could be pasted into.

On Fri, Aug 30, 2019 at 9:05 PM Bryan Bende <bb...@gmail.com> wrote:
>
> I’d like to see us move towards the versioned flow representation from
> registry and add features to NiFi like right click on PG and export to
> versioned flow, import from a file etc.
>
> The Minifi java toolkit already has a transform from versioned flow JSON to
> YAML, just like the template transform.
>
>
> On Fri, Aug 30, 2019 at 4:05 PM Samuel Hjelmfelt
> <sa...@yahoo.com.invalid> wrote:
>
> > Hello everyone,NIFI-6539 brings up the topic of deployment scenarios when
> > a NiFi registry is not available. MiNiFi allows configuration via a YML
> > file-based representation of a flow, often created from an exported NiFi
> > template. However, there has been a push to focus on versioned flows in the
> > registry and to move away from templates.
> > Long term, what is the plan for file-based flow representations? Is the
> > intent to continue using templates even in NiFi 2.x, or is there some other
> > solution planned? Both deployments without a registry and registry
> > import/export will require some kind of file-based flow representation.
> > -Sam
> >
> --
> Sent from Gmail Mobile

Re: File-based Versioned Flow Representation

[Bryan Bende <bb...@gmail.com>]

I’d like to see us move towards the versioned flow representation from
registry and add features to NiFi like right click on PG and export to
versioned flow, import from a file etc.

The Minifi java toolkit already has a transform from versioned flow JSON to
YAML, just like the template transform.


On Fri, Aug 30, 2019 at 4:05 PM Samuel Hjelmfelt
<sa...@yahoo.com.invalid> wrote:

> Hello everyone,NIFI-6539 brings up the topic of deployment scenarios when
> a NiFi registry is not available. MiNiFi allows configuration via a YML
> file-based representation of a flow, often created from an exported NiFi
> template. However, there has been a push to focus on versioned flows in the
> registry and to move away from templates.
> Long term, what is the plan for file-based flow representations? Is the
> intent to continue using templates even in NiFi 2.x, or is there some other
> solution planned? Both deployments without a registry and registry
> import/export will require some kind of file-based flow representation.
> -Sam
>
-- 
Sent from Gmail Mobile