You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hop.apache.org by Matt Casters <ma...@neo4j.com.INVALID> on 2021/02/02 11:57:52 UTC
[DISCUSS] Plugin samples
Hi Hoppers,
It took a bit to figure this out but PR 588 now includes an automatic way
of picking up all 'src/main/samples' folders in the various plugins in our
wonderful project. Thanks for the feedback so far.
The 'samples' project is the companion of the 'default' project and is
pretty much coming along as discussed earlier.
I wrote some docs
<https://github.com/apache/incubator-hop-docs/blob/cb83f1db498f26c6f93bd51dc25fc64948241a8f/hop-dev-manual/modules/ROOT/pages/plugin-samples.adoc>
on how to include samples. (Please merge along with PR588)
What is needed are suggestions as to how the folder structure should look
like when all is said and done. I'm strongly in favor of keeping things
simple with the naming standard:
Folders:
* transforms/ : transform examples
* actions/ : action examples
* beam/ : beam examples
File naming:
<name of the transform/action> - <what it is demoed>.<hpl/hwf>
If we could reach some sort of consensus I can add this to the
documentation.
Perhaps as a minimum we could have a note in the pipeline or workflow
describing what the sample in question does? Other ideas?
Cheers,
Matt
Re: [DISCUSS] Plugin samples
Posted by "Sergio Ramazzina (SERASOFT)" <se...@serasoft.it>.
Ciao Hans,
I'm sorry but I read too fast Matt's documentation where the idea of
this naming convention was very cleary specified. Thanks for pointing me
there. The only drawback about this is that we must remember to name the
files this way. But apart from this I'm fine with that.
I just suggest to explore the idea of writing something that enforces
this naming when we are going to get the files from plugins' sample
subdir to copy them under the samples project. Writing a new maven
plugin that, before copying the file to the sample project's dir, checks
for the naming and eventually on fly appends to sample's file name the
plugin's name according to Matt proposal would be a good idea.
Cheers
Sergio
Il 03/02/2021 10:04, Hans Van Akelyen ha scritto:
> Hi Sergio,
>
> I am not sure if a user is waiting to see a samples project with a
> "transforms" folder and under that folder 300 sub directories each
> containing 1 or 2 pipelines.
> Adding the transform name to the .hpl file seems to be sufficient.
>
> Cheers,
> Hans
>
> On Wed, Feb 3, 2021 at 9:36 AM Sergio Ramazzina (SERASOFT) <
> sergio.ramazzina@serasoft.it> wrote:
>
>> Hi All,
>>
>> I'm also in favour of simplicity. Just an idea, would it be a good
>> thing, under the three main dirs Matt proposed, to keep files separated
>> by plugin name and eventually have a common directory to place all the
>> files needed by the samples to run properly? Having a set of
>> subdirectories named like plugin's name will prevent files to be
>> overwritten.
>>
>> Cheers
>>
>> Sergio
>>
>> Il 02/02/2021 22:51, Hans Van Akelyen ha scritto:
>>> Hi All,
>>>
>>> Totally agree that we should try and keep this as simple as possible.
>>> The naming convention you are suggesting is simple, I suggest we do the
>>> same for input files or special metadata objects that are needed.
>>>
>>> One thing to keep in mind is that the files with the same name will be
>>> overwritten, maybe we should see if we can make the build fail or add a
>>> check to our PR flow.
>>> The current PR flow does not go to the assembly phase so even if we make
>>> maven fail we will only see it in Jenkins
>>>
>>> Cheers,
>>> Hans
>>>
>>> On Tue, Feb 2, 2021 at 12:58 PM Matt Casters <matt.casters@neo4j.com
>> .invalid>
>>> wrote:
>>>
>>>> Hi Hoppers,
>>>>
>>>> It took a bit to figure this out but PR 588 now includes an automatic
>> way
>>>> of picking up all 'src/main/samples' folders in the various plugins in
>> our
>>>> wonderful project. Thanks for the feedback so far.
>>>>
>>>> The 'samples' project is the companion of the 'default' project and is
>>>> pretty much coming along as discussed earlier.
>>>>
>>>> I wrote some docs
>>>> <
>>>>
>> https://github.com/apache/incubator-hop-docs/blob/cb83f1db498f26c6f93bd51dc25fc64948241a8f/hop-dev-manual/modules/ROOT/pages/plugin-samples.adoc
>>>> on how to include samples. (Please merge along with PR588)
>>>>
>>>> What is needed are suggestions as to how the folder structure should
>> look
>>>> like when all is said and done. I'm strongly in favor of keeping things
>>>> simple with the naming standard:
>>>>
>>>> Folders:
>>>> * transforms/ : transform examples
>>>> * actions/ : action examples
>>>> * beam/ : beam examples
>>>>
>>>> File naming:
>>>> <name of the transform/action> - <what it is demoed>.<hpl/hwf>
>>>>
>>>> If we could reach some sort of consensus I can add this to the
>>>> documentation.
>>>> Perhaps as a minimum we could have a note in the pipeline or workflow
>>>> describing what the sample in question does? Other ideas?
>>>>
>>>> Cheers,
>>>> Matt
>>>>
>> --
>>
Re: [DISCUSS] Plugin samples
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi Sergio,
I am not sure if a user is waiting to see a samples project with a
"transforms" folder and under that folder 300 sub directories each
containing 1 or 2 pipelines.
Adding the transform name to the .hpl file seems to be sufficient.
Cheers,
Hans
On Wed, Feb 3, 2021 at 9:36 AM Sergio Ramazzina (SERASOFT) <
sergio.ramazzina@serasoft.it> wrote:
> Hi All,
>
> I'm also in favour of simplicity. Just an idea, would it be a good
> thing, under the three main dirs Matt proposed, to keep files separated
> by plugin name and eventually have a common directory to place all the
> files needed by the samples to run properly? Having a set of
> subdirectories named like plugin's name will prevent files to be
> overwritten.
>
> Cheers
>
> Sergio
>
> Il 02/02/2021 22:51, Hans Van Akelyen ha scritto:
> > Hi All,
> >
> > Totally agree that we should try and keep this as simple as possible.
> > The naming convention you are suggesting is simple, I suggest we do the
> > same for input files or special metadata objects that are needed.
> >
> > One thing to keep in mind is that the files with the same name will be
> > overwritten, maybe we should see if we can make the build fail or add a
> > check to our PR flow.
> > The current PR flow does not go to the assembly phase so even if we make
> > maven fail we will only see it in Jenkins
> >
> > Cheers,
> > Hans
> >
> > On Tue, Feb 2, 2021 at 12:58 PM Matt Casters <matt.casters@neo4j.com
> .invalid>
> > wrote:
> >
> >> Hi Hoppers,
> >>
> >> It took a bit to figure this out but PR 588 now includes an automatic
> way
> >> of picking up all 'src/main/samples' folders in the various plugins in
> our
> >> wonderful project. Thanks for the feedback so far.
> >>
> >> The 'samples' project is the companion of the 'default' project and is
> >> pretty much coming along as discussed earlier.
> >>
> >> I wrote some docs
> >> <
> >>
> https://github.com/apache/incubator-hop-docs/blob/cb83f1db498f26c6f93bd51dc25fc64948241a8f/hop-dev-manual/modules/ROOT/pages/plugin-samples.adoc
> >> on how to include samples. (Please merge along with PR588)
> >>
> >> What is needed are suggestions as to how the folder structure should
> look
> >> like when all is said and done. I'm strongly in favor of keeping things
> >> simple with the naming standard:
> >>
> >> Folders:
> >> * transforms/ : transform examples
> >> * actions/ : action examples
> >> * beam/ : beam examples
> >>
> >> File naming:
> >> <name of the transform/action> - <what it is demoed>.<hpl/hwf>
> >>
> >> If we could reach some sort of consensus I can add this to the
> >> documentation.
> >> Perhaps as a minimum we could have a note in the pipeline or workflow
> >> describing what the sample in question does? Other ideas?
> >>
> >> Cheers,
> >> Matt
> >>
> --
>
Re: [DISCUSS] Plugin samples
Posted by "Sergio Ramazzina (SERASOFT)" <se...@serasoft.it>.
Hi All,
I'm also in favour of simplicity. Just an idea, would it be a good
thing, under the three main dirs Matt proposed, to keep files separated
by plugin name and eventually have a common directory to place all the
files needed by the samples to run properly? Having a set of
subdirectories named like plugin's name will prevent files to be
overwritten.
Cheers
Sergio
Il 02/02/2021 22:51, Hans Van Akelyen ha scritto:
> Hi All,
>
> Totally agree that we should try and keep this as simple as possible.
> The naming convention you are suggesting is simple, I suggest we do the
> same for input files or special metadata objects that are needed.
>
> One thing to keep in mind is that the files with the same name will be
> overwritten, maybe we should see if we can make the build fail or add a
> check to our PR flow.
> The current PR flow does not go to the assembly phase so even if we make
> maven fail we will only see it in Jenkins
>
> Cheers,
> Hans
>
> On Tue, Feb 2, 2021 at 12:58 PM Matt Casters <ma...@neo4j.com.invalid>
> wrote:
>
>> Hi Hoppers,
>>
>> It took a bit to figure this out but PR 588 now includes an automatic way
>> of picking up all 'src/main/samples' folders in the various plugins in our
>> wonderful project. Thanks for the feedback so far.
>>
>> The 'samples' project is the companion of the 'default' project and is
>> pretty much coming along as discussed earlier.
>>
>> I wrote some docs
>> <
>> https://github.com/apache/incubator-hop-docs/blob/cb83f1db498f26c6f93bd51dc25fc64948241a8f/hop-dev-manual/modules/ROOT/pages/plugin-samples.adoc
>> on how to include samples. (Please merge along with PR588)
>>
>> What is needed are suggestions as to how the folder structure should look
>> like when all is said and done. I'm strongly in favor of keeping things
>> simple with the naming standard:
>>
>> Folders:
>> * transforms/ : transform examples
>> * actions/ : action examples
>> * beam/ : beam examples
>>
>> File naming:
>> <name of the transform/action> - <what it is demoed>.<hpl/hwf>
>>
>> If we could reach some sort of consensus I can add this to the
>> documentation.
>> Perhaps as a minimum we could have a note in the pipeline or workflow
>> describing what the sample in question does? Other ideas?
>>
>> Cheers,
>> Matt
>>
--
Re: [DISCUSS] Plugin samples
Posted by Matt Casters <ma...@neo4j.com.INVALID>.
Or we can just add to the documentation: don't use the same file twice ;-)
On Tue, Feb 2, 2021 at 10:51 PM Hans Van Akelyen <ha...@gmail.com>
wrote:
> Hi All,
>
> Totally agree that we should try and keep this as simple as possible.
> The naming convention you are suggesting is simple, I suggest we do the
> same for input files or special metadata objects that are needed.
>
> One thing to keep in mind is that the files with the same name will be
> overwritten, maybe we should see if we can make the build fail or add a
> check to our PR flow.
> The current PR flow does not go to the assembly phase so even if we make
> maven fail we will only see it in Jenkins
>
> Cheers,
> Hans
>
> On Tue, Feb 2, 2021 at 12:58 PM Matt Casters <matt.casters@neo4j.com
> .invalid>
> wrote:
>
> > Hi Hoppers,
> >
> > It took a bit to figure this out but PR 588 now includes an automatic way
> > of picking up all 'src/main/samples' folders in the various plugins in
> our
> > wonderful project. Thanks for the feedback so far.
> >
> > The 'samples' project is the companion of the 'default' project and is
> > pretty much coming along as discussed earlier.
> >
> > I wrote some docs
> > <
> >
> https://github.com/apache/incubator-hop-docs/blob/cb83f1db498f26c6f93bd51dc25fc64948241a8f/hop-dev-manual/modules/ROOT/pages/plugin-samples.adoc
> > >
> > on how to include samples. (Please merge along with PR588)
> >
> > What is needed are suggestions as to how the folder structure should look
> > like when all is said and done. I'm strongly in favor of keeping things
> > simple with the naming standard:
> >
> > Folders:
> > * transforms/ : transform examples
> > * actions/ : action examples
> > * beam/ : beam examples
> >
> > File naming:
> > <name of the transform/action> - <what it is demoed>.<hpl/hwf>
> >
> > If we could reach some sort of consensus I can add this to the
> > documentation.
> > Perhaps as a minimum we could have a note in the pipeline or workflow
> > describing what the sample in question does? Other ideas?
> >
> > Cheers,
> > Matt
> >
>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
☎ +32486972937
Re: [DISCUSS] Plugin samples
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi All,
Totally agree that we should try and keep this as simple as possible.
The naming convention you are suggesting is simple, I suggest we do the
same for input files or special metadata objects that are needed.
One thing to keep in mind is that the files with the same name will be
overwritten, maybe we should see if we can make the build fail or add a
check to our PR flow.
The current PR flow does not go to the assembly phase so even if we make
maven fail we will only see it in Jenkins
Cheers,
Hans
On Tue, Feb 2, 2021 at 12:58 PM Matt Casters <ma...@neo4j.com.invalid>
wrote:
> Hi Hoppers,
>
> It took a bit to figure this out but PR 588 now includes an automatic way
> of picking up all 'src/main/samples' folders in the various plugins in our
> wonderful project. Thanks for the feedback so far.
>
> The 'samples' project is the companion of the 'default' project and is
> pretty much coming along as discussed earlier.
>
> I wrote some docs
> <
> https://github.com/apache/incubator-hop-docs/blob/cb83f1db498f26c6f93bd51dc25fc64948241a8f/hop-dev-manual/modules/ROOT/pages/plugin-samples.adoc
> >
> on how to include samples. (Please merge along with PR588)
>
> What is needed are suggestions as to how the folder structure should look
> like when all is said and done. I'm strongly in favor of keeping things
> simple with the naming standard:
>
> Folders:
> * transforms/ : transform examples
> * actions/ : action examples
> * beam/ : beam examples
>
> File naming:
> <name of the transform/action> - <what it is demoed>.<hpl/hwf>
>
> If we could reach some sort of consensus I can add this to the
> documentation.
> Perhaps as a minimum we could have a note in the pipeline or workflow
> describing what the sample in question does? Other ideas?
>
> Cheers,
> Matt
>