New Enhanced JSON Output Transform

["Sergio Ramazzina (SERASOFT)" <se...@serasoft.it>]

Dear all,

yesterday I released a PR containing the new Enhanced JSON Output 
Transform I ported from our Kettle plugins to Hop. It is a little 
different and more powerful respect to the current JSON Output transform 
and was written mainly to overcome its limitations. I'm going to write 
the needed documentation and build some samples that I will push as soon 
as possible so that anyone will be able to try it out. The new transform 
is completely based on Jackson to process JSON files. All the new 
functionality like Pritty Printing and Injection has been applied to 
this new Transform leaving the original unchanged.

Would be nice in my opinion to introduce a new category of the Transform 
selection dialog that we can call Preview and put in there all the new 
Transforms that we decide to include in Hop. They will be considereded 
more explicitly as "Beta Versions" until we vote to promote the new 
Transform as "Production Ready" or "Certified" and move them in the real 
categories.

Cheers

Sergio


-- 

Re: New Enhanced JSON Output Transform

[Hans Van Akelyen <ha...@gmail.com>]

Hi All,

Thanks for the input! I have created a general Jira case [1] comments or
extra tasks can be added to this case.

As Nicolas stated, the problem can currently be solved by using the
categories.
Using the deprecated category will give a clear indication in the GUI that
the transform is no longer supported
For new actions or transforms the best way to do it currently would be to
use the BaseTransform.Category.Experimental or create a new category


Cheers,
Hans

[1] https://issues.apache.org/jira/browse/HOP-2424

On Thu, Jan 14, 2021 at 9:50 PM Brandon Jackson <us...@gmail.com> wrote:

> I like the idea of these annotations.  It might be interesting if we could
> collect and emit telemetry data about steps used too.
> I could imagine scanning my projects to get an idea of how conformant it is
> with current standards.
> If pipelines are in production; there is some implied engineering effort to
> switch out deprecated steps to maintained steps that we would want to
> enable ETL designers with.
> The annotations will help a lot; but what should the policy be.  Next
> release deprecated plugins are gone; or after a major release?
>
> What criteria positions a plugin for deprecation?
>
> Brandon
>
>
>
> On Thu, Jan 14, 2021 at 9:25 AM Sergio Ramazzina (SERASOFT) <
> sergio.ramazzina@serasoft.it> wrote:
>
> > Below my comments:
> >
> > Il 14/01/2021 10:34, Matt Casters ha scritto:
> > > Great ideas. Having indicators for users to let them know the state of
> a
> > > plugin is a great way of going about things I think.
> > > That being said I think that terms like "production ready" are
> ambiguous
> > > and rather arbitrary.
> > Yes Matt you are right I misused that term, it was better to say
> > something like just "stable".
> > > Perhaps we can put some additional requirements on that?
> > >
> > > Perhaps we should require for "production ready" plugins the following
> to
> > > be available:
> > > - Documentation
> > > - Integration test(s)
> > > - Sample(s)
> > >
> > > Perhaps this is also more an indication of stability, rather than
> > > production readiness?
> > > If that's the case we come to something like this:
> > >
> > > * *Stable* : Documented, multiple integration tests (IT) & Samples
> > > * *Beta* : first introduced version, no IT, docs or samples
> > > * *Alpha* : experimental
> > > * *Deprecated* : will be removed in a future version
> > >
> > > For example, right now only a few transforms would fall under Stable.
> > > If we could set up rules and definitions like this I think it would
> > really
> > > help us as well to assess how far along we are towards our goal of
> > hitting
> > > a 1.0 with only "stable" plugins in Hop.
> >
> > Would be nice to have a mechanism to check that all of the required
> > parts (Documentation, Integrations test, Samples) are present and that
> > integration tests passed without any problems and basing on the results
> > of these checks, flags the Transform as Stable, Beta and Alpha
> > automatically. Deprecation is the only thing requires to be managed
> > manually
> >
> > Cheers
> >
> > S
> >
> >
>

Re: New Enhanced JSON Output Transform

[Brandon Jackson <us...@gmail.com>]

I like the idea of these annotations.  It might be interesting if we could
collect and emit telemetry data about steps used too.
I could imagine scanning my projects to get an idea of how conformant it is
with current standards.
If pipelines are in production; there is some implied engineering effort to
switch out deprecated steps to maintained steps that we would want to
enable ETL designers with.
The annotations will help a lot; but what should the policy be.  Next
release deprecated plugins are gone; or after a major release?

What criteria positions a plugin for deprecation?

Brandon



On Thu, Jan 14, 2021 at 9:25 AM Sergio Ramazzina (SERASOFT) <
sergio.ramazzina@serasoft.it> wrote:

> Below my comments:
>
> Il 14/01/2021 10:34, Matt Casters ha scritto:
> > Great ideas. Having indicators for users to let them know the state of a
> > plugin is a great way of going about things I think.
> > That being said I think that terms like "production ready" are ambiguous
> > and rather arbitrary.
> Yes Matt you are right I misused that term, it was better to say
> something like just "stable".
> > Perhaps we can put some additional requirements on that?
> >
> > Perhaps we should require for "production ready" plugins the following to
> > be available:
> > - Documentation
> > - Integration test(s)
> > - Sample(s)
> >
> > Perhaps this is also more an indication of stability, rather than
> > production readiness?
> > If that's the case we come to something like this:
> >
> > * *Stable* : Documented, multiple integration tests (IT) & Samples
> > * *Beta* : first introduced version, no IT, docs or samples
> > * *Alpha* : experimental
> > * *Deprecated* : will be removed in a future version
> >
> > For example, right now only a few transforms would fall under Stable.
> > If we could set up rules and definitions like this I think it would
> really
> > help us as well to assess how far along we are towards our goal of
> hitting
> > a 1.0 with only "stable" plugins in Hop.
>
> Would be nice to have a mechanism to check that all of the required
> parts (Documentation, Integrations test, Samples) are present and that
> integration tests passed without any problems and basing on the results
> of these checks, flags the Transform as Stable, Beta and Alpha
> automatically. Deprecation is the only thing requires to be managed
> manually
>
> Cheers
>
> S
>
>

Re: New Enhanced JSON Output Transform

["Sergio Ramazzina (SERASOFT)" <se...@serasoft.it>]

Below my comments:

Il 14/01/2021 10:34, Matt Casters ha scritto:
> Great ideas. Having indicators for users to let them know the state of a
> plugin is a great way of going about things I think.
> That being said I think that terms like "production ready" are ambiguous
> and rather arbitrary.
Yes Matt you are right I misused that term, it was better to say 
something like just "stable".
> Perhaps we can put some additional requirements on that?
>
> Perhaps we should require for "production ready" plugins the following to
> be available:
> - Documentation
> - Integration test(s)
> - Sample(s)
>
> Perhaps this is also more an indication of stability, rather than
> production readiness?
> If that's the case we come to something like this:
>
> * *Stable* : Documented, multiple integration tests (IT) & Samples
> * *Beta* : first introduced version, no IT, docs or samples
> * *Alpha* : experimental
> * *Deprecated* : will be removed in a future version
>
> For example, right now only a few transforms would fall under Stable.
> If we could set up rules and definitions like this I think it would really
> help us as well to assess how far along we are towards our goal of hitting
> a 1.0 with only "stable" plugins in Hop.

Would be nice to have a mechanism to check that all of the required 
parts (Documentation, Integrations test, Samples) are present and that 
integration tests passed without any problems and basing on the results 
of these checks, flags the Transform as Stable, Beta and Alpha 
automatically. Deprecation is the only thing requires to be managed manually

Cheers

S

Re: New Enhanced JSON Output Transform

[Matt Casters <ma...@neo4j.com.INVALID>]

Great ideas. Having indicators for users to let them know the state of a
plugin is a great way of going about things I think.
That being said I think that terms like "production ready" are ambiguous
and rather arbitrary.
Perhaps we can put some additional requirements on that?

Perhaps we should require for "production ready" plugins the following to
be available:
- Documentation
- Integration test(s)
- Sample(s)

Perhaps this is also more an indication of stability, rather than
production readiness?
If that's the case we come to something like this:

* *Stable* : Documented, multiple integration tests (IT) & Samples
* *Beta* : first introduced version, no IT, docs or samples
* *Alpha* : experimental
* *Deprecated* : will be removed in a future version

For example, right now only a few transforms would fall under Stable.
If we could set up rules and definitions like this I think it would really
help us as well to assess how far along we are towards our goal of hitting
a 1.0 with only "stable" plugins in Hop.

Cheers,
Matt


On Wed, Jan 13, 2021 at 2:55 PM Nicolas ADMENT <na...@gmail.com> wrote:

> Hello,
>
> I am also of the opinion to add a status to the annotation of the Action /
> Transform plugins.
>
>    - *PRODUCTION *by default if not specified and not displayed anywhere
>    - *BETA *displayed in the selection dialog box, and why not on the graph
>    - *DEPRECATED *displayed in the selection dialog box and on the graph
>    (already the case)
>
> Cheers
>
> Nicolas
>
> Le mer. 13 janv. 2021 à 12:18, Sergio Ramazzina (SERASOFT) <
> sergio.ramazzina@serasoft.it> a écrit :
>
> > Ciao Hans,
> >
> > I really like your idea about having a visual indicator to explicitly
> > indicate experimental/deprecated Transforms so that we can be more
> > explicit on this at any time. Would be nice if that indicator will be
> > visible both in the Selection Dialog (when choosing a Transform that is
> > needed for our pipeline) and in the canvas. Extending the semantic of
> > plugin's annotations with a new attribute for this is fine because will
> > be the natural and easier way for a programmer to specify this
> > characteristic of a transform.
> >
> > At this point, we can apply the same approach to Workflow Activities.
> >
> > Cheers,
> >
> > Sergio
> >
> > Il 13/01/2021 10:53, Hans Van Akelyen ha scritto:
> > > Hi Sergio,
> > >
> > > I think adding an extra category is a great idea but this will only
> help
> > > during the development of a pipeline or workflow.
> > > Another extra option we could consider is adding a visual indicator to
> > the
> > > transform/action on the canvas.
> > > on the right top we add a (v) or (x) when a transform has finished
> > running
> > > but maybe we can add an indicator to the left top to indicate that a
> > > transform is experimental or deprecated in the current version of Hop.
> > >
> > > We can extend the plugin annotation to indicate the "state" of the
> > plugin.
> > > In the future we could add extra information to that left top corner.
> > > for example, indicate that you are using a plugin that wasn't developed
> > for
> > > that version of Hop
> > >
> > > Would love to hear everyone's opinion on this idea.
> > >
> > > Cheers,
> > > Hans
> > >
> > >
> > > On Wed, Jan 13, 2021 at 10:00 AM Sergio Ramazzina (SERASOFT) <
> > > sergio.ramazzina@serasoft.it> wrote:
> > >
> > >> Dear all,
> > >>
> > >> yesterday I released a PR containing the new Enhanced JSON Output
> > >> Transform I ported from our Kettle plugins to Hop. It is a little
> > >> different and more powerful respect to the current JSON Output
> transform
> > >> and was written mainly to overcome its limitations. I'm going to write
> > >> the needed documentation and build some samples that I will push as
> soon
> > >> as possible so that anyone will be able to try it out. The new
> transform
> > >> is completely based on Jackson to process JSON files. All the new
> > >> functionality like Pritty Printing and Injection has been applied to
> > >> this new Transform leaving the original unchanged.
> > >>
> > >> Would be nice in my opinion to introduce a new category of the
> Transform
> > >> selection dialog that we can call Preview and put in there all the new
> > >> Transforms that we decide to include in Hop. They will be considereded
> > >> more explicitly as "Beta Versions" until we vote to promote the new
> > >> Transform as "Production Ready" or "Certified" and move them in the
> real
> > >> categories.
> > >>
> > >> Cheers
> > >>
> > >> Sergio
> > >>
> > >>
> > >> --
> > >>
> > --
> >
>


-- 
Neo4j Chief Solutions Architect
*✉   *matt.casters@neo4j.com
☎  +32486972937

Re: New Enhanced JSON Output Transform

[Nicolas ADMENT <na...@gmail.com>]

Hello,

I am also of the opinion to add a status to the annotation of the Action /
Transform plugins.

   - *PRODUCTION *by default if not specified and not displayed anywhere
   - *BETA *displayed in the selection dialog box, and why not on the graph
   - *DEPRECATED *displayed in the selection dialog box and on the graph
   (already the case)

Cheers

Nicolas

Le mer. 13 janv. 2021 à 12:18, Sergio Ramazzina (SERASOFT) <
sergio.ramazzina@serasoft.it> a écrit :

> Ciao Hans,
>
> I really like your idea about having a visual indicator to explicitly
> indicate experimental/deprecated Transforms so that we can be more
> explicit on this at any time. Would be nice if that indicator will be
> visible both in the Selection Dialog (when choosing a Transform that is
> needed for our pipeline) and in the canvas. Extending the semantic of
> plugin's annotations with a new attribute for this is fine because will
> be the natural and easier way for a programmer to specify this
> characteristic of a transform.
>
> At this point, we can apply the same approach to Workflow Activities.
>
> Cheers,
>
> Sergio
>
> Il 13/01/2021 10:53, Hans Van Akelyen ha scritto:
> > Hi Sergio,
> >
> > I think adding an extra category is a great idea but this will only help
> > during the development of a pipeline or workflow.
> > Another extra option we could consider is adding a visual indicator to
> the
> > transform/action on the canvas.
> > on the right top we add a (v) or (x) when a transform has finished
> running
> > but maybe we can add an indicator to the left top to indicate that a
> > transform is experimental or deprecated in the current version of Hop.
> >
> > We can extend the plugin annotation to indicate the "state" of the
> plugin.
> > In the future we could add extra information to that left top corner.
> > for example, indicate that you are using a plugin that wasn't developed
> for
> > that version of Hop
> >
> > Would love to hear everyone's opinion on this idea.
> >
> > Cheers,
> > Hans
> >
> >
> > On Wed, Jan 13, 2021 at 10:00 AM Sergio Ramazzina (SERASOFT) <
> > sergio.ramazzina@serasoft.it> wrote:
> >
> >> Dear all,
> >>
> >> yesterday I released a PR containing the new Enhanced JSON Output
> >> Transform I ported from our Kettle plugins to Hop. It is a little
> >> different and more powerful respect to the current JSON Output transform
> >> and was written mainly to overcome its limitations. I'm going to write
> >> the needed documentation and build some samples that I will push as soon
> >> as possible so that anyone will be able to try it out. The new transform
> >> is completely based on Jackson to process JSON files. All the new
> >> functionality like Pritty Printing and Injection has been applied to
> >> this new Transform leaving the original unchanged.
> >>
> >> Would be nice in my opinion to introduce a new category of the Transform
> >> selection dialog that we can call Preview and put in there all the new
> >> Transforms that we decide to include in Hop. They will be considereded
> >> more explicitly as "Beta Versions" until we vote to promote the new
> >> Transform as "Production Ready" or "Certified" and move them in the real
> >> categories.
> >>
> >> Cheers
> >>
> >> Sergio
> >>
> >>
> >> --
> >>
> --
>

Re: New Enhanced JSON Output Transform

["Sergio Ramazzina (SERASOFT)" <se...@serasoft.it>]

Ciao Hans,

I really like your idea about having a visual indicator to explicitly 
indicate experimental/deprecated Transforms so that we can be more 
explicit on this at any time. Would be nice if that indicator will be 
visible both in the Selection Dialog (when choosing a Transform that is 
needed for our pipeline) and in the canvas. Extending the semantic of 
plugin's annotations with a new attribute for this is fine because will 
be the natural and easier way for a programmer to specify this 
characteristic of a transform.

At this point, we can apply the same approach to Workflow Activities.

Cheers,

Sergio

Il 13/01/2021 10:53, Hans Van Akelyen ha scritto:
> Hi Sergio,
>
> I think adding an extra category is a great idea but this will only help
> during the development of a pipeline or workflow.
> Another extra option we could consider is adding a visual indicator to the
> transform/action on the canvas.
> on the right top we add a (v) or (x) when a transform has finished running
> but maybe we can add an indicator to the left top to indicate that a
> transform is experimental or deprecated in the current version of Hop.
>
> We can extend the plugin annotation to indicate the "state" of the plugin.
> In the future we could add extra information to that left top corner.
> for example, indicate that you are using a plugin that wasn't developed for
> that version of Hop
>
> Would love to hear everyone's opinion on this idea.
>
> Cheers,
> Hans
>
>
> On Wed, Jan 13, 2021 at 10:00 AM Sergio Ramazzina (SERASOFT) <
> sergio.ramazzina@serasoft.it> wrote:
>
>> Dear all,
>>
>> yesterday I released a PR containing the new Enhanced JSON Output
>> Transform I ported from our Kettle plugins to Hop. It is a little
>> different and more powerful respect to the current JSON Output transform
>> and was written mainly to overcome its limitations. I'm going to write
>> the needed documentation and build some samples that I will push as soon
>> as possible so that anyone will be able to try it out. The new transform
>> is completely based on Jackson to process JSON files. All the new
>> functionality like Pritty Printing and Injection has been applied to
>> this new Transform leaving the original unchanged.
>>
>> Would be nice in my opinion to introduce a new category of the Transform
>> selection dialog that we can call Preview and put in there all the new
>> Transforms that we decide to include in Hop. They will be considereded
>> more explicitly as "Beta Versions" until we vote to promote the new
>> Transform as "Production Ready" or "Certified" and move them in the real
>> categories.
>>
>> Cheers
>>
>> Sergio
>>
>>
>> --
>>
-- 

Re: New Enhanced JSON Output Transform

[Hans Van Akelyen <ha...@gmail.com>]

Hi Sergio,

I think adding an extra category is a great idea but this will only help
during the development of a pipeline or workflow.
Another extra option we could consider is adding a visual indicator to the
transform/action on the canvas.
on the right top we add a (v) or (x) when a transform has finished running
but maybe we can add an indicator to the left top to indicate that a
transform is experimental or deprecated in the current version of Hop.

We can extend the plugin annotation to indicate the "state" of the plugin.
In the future we could add extra information to that left top corner.
for example, indicate that you are using a plugin that wasn't developed for
that version of Hop

Would love to hear everyone's opinion on this idea.

Cheers,
Hans


On Wed, Jan 13, 2021 at 10:00 AM Sergio Ramazzina (SERASOFT) <
sergio.ramazzina@serasoft.it> wrote:

> Dear all,
>
> yesterday I released a PR containing the new Enhanced JSON Output
> Transform I ported from our Kettle plugins to Hop. It is a little
> different and more powerful respect to the current JSON Output transform
> and was written mainly to overcome its limitations. I'm going to write
> the needed documentation and build some samples that I will push as soon
> as possible so that anyone will be able to try it out. The new transform
> is completely based on Jackson to process JSON files. All the new
> functionality like Pritty Printing and Injection has been applied to
> this new Transform leaving the original unchanged.
>
> Would be nice in my opinion to introduce a new category of the Transform
> selection dialog that we can call Preview and put in there all the new
> Transforms that we decide to include in Hop. They will be considereded
> more explicitly as "Beta Versions" until we vote to promote the new
> Transform as "Production Ready" or "Certified" and move them in the real
> categories.
>
> Cheers
>
> Sergio
>
>
> --
>