You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hop.apache.org by Matt Casters <ma...@neo4j.com.INVALID> on 2021/06/23 16:48:07 UTC
[DISCUSS] Imported source code
Hello Devel-hoppers,
Over the course of the last couple of months we've imported some additional
3rd party source code.
As was mentioned earlier during release voting we need to do a better job
of making sure that we have permission to donate said code to Apache.
For everything related to my own code and the Neo4j plugins I can be formal
the following are donated:
- Code we imported: kettle-beam, pentaho-pdi-dataset, kettle-debug-plugin,
kettle-needful-things, kettle-environment, kettle-splunk, kettle-kafka
- Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens wrote
the original version of that code so if he agrees to donate that code we're
fine there.
Obviously for as far as myself is concerned: everything I added to the
incubator-hop source code is also covered by my ICLA and a number of
sources were mentioned in the incubation proposal.
I'm just trying to be complete since apparently I've been a proficient
coder lately :-)
Recently we removed some old SQL parser code (good riddance).
Since everything in Hop is a plugin anyway I suggest that if we're even a
little bit uncertain about the origin of the plugin source code (apart from
the original source code import/rewrite) that we simply move it out into
the external plugins <https://github.com/project-hop/hop-plugins> until we
have clear permission of the original authors to use the code. Going
forward I suggest we do the opposite. Let's add code to the external
plugins project first and only then incorporate it into incubator-hop when
we have a more formal permission to donate the code to the ASF.
I believe that the list of debatable plugins is this:
- MQTT input/output (originally from Hitachi)
- Google Sheets (informal permission received from JF Monteil, need to
formalize)
- Google Analytics (originally from Hitachi)
- Dropbox (permission needed from L. Coelho)
The other new plugins were written from scratch and are fine I believe.
Personally I think it's just not worth it to even have these discussions
for a couple of obscure plugins so I would suggest moving this code out of
our way until we have clear SGAs for these plugins.
Let me know what your thoughts are! Perhaps if we have more external
plugins it forces us to build a software marketplace at some point.
Question for our mentors: in what forms do these agreements need to come?
Where are they stored and how are they communicated?
Cheers,
Matt
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
Re: [DISCUSS] Imported source code
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi All,
I have removed the following plugins to unblock the 0.99 release:
- MQTT input/output (originally from Hitachi)
- Google Sheets (informal permission received from JF Monteil, need to
formalize)
- Google Analytics (originally from Hitachi)
- Dropbox (permission needed from L. Coelho)
I do not think other issues were raised, I will wait another 24 hours and
then start working on a new release candidate.
Cheers,
Hans
On Fri, 25 Jun 2021 at 14:35, Matt Casters <ma...@neo4j.com.invalid>
wrote:
> I think in that case perhaps we can create an incubator-hop-external
> repository or something like that which we don't release. We can build it
> and perhaps even put it up somewhere.
> The goal of that repository would be to evaluate licenses and
> code-contributions. In case it's clear we're dealing with a conflicting
> license the code can live in project-hop/hop-plugins or indeed anywhere
> else.
>
> In general I'd be quite happy with this state of affairs as it offers a
> clear path into code contributions which we can document. Besides license
> and copyright I would add other requirements like the need to have
> integration tests, documentation and samples for the plugins.
>
> I'll start a different discussion thread on the "marketplace" idea since I
> think it'll become more important. There are already companies and
> organizations porting their code and functionality over to the Apache Hop
> API so we shouldn't wait too long to make a decision.
>
> Cheers,
> Matt
>
>
>
> On Thu, Jun 24, 2021 at 10:22 PM Julian Hyde <jh...@gmail.com>
> wrote:
>
> > Apache policy doesn’t really think in terms of Git repositories. An
> Apache
> > PMC releases source code as a signed tar ball. (The PMC needs to be able
> to
> > guarantee the provenance of that code, so in practice of course there is
> a
> > source control system such as Git or Subversion.)
> >
> > If this plugin code is going to be released by the PMC, it must be under
> a
> > "category A" license [2], which means Apache or another permissive
> license
> > such as MIT or BSD. Such code can live in our source control system, once
> > it has passed IP clearance.
> >
> > EPL is category B, which means that an Apache project can distribute
> > binaries but not source code.
> >
> > GPL and LGPL are category X, which means that we cannot distribute
> > binaries or source code. We can depend on a category X component (say a
> > JDBC driver) via a plugin API as long as it isn’t the only possible
> > implementation of that API.
> >
> > So, to answer your question: an Apache project cannot have GPL or LGPL
> > repositories. There would be no point, because it cannot release GPL or
> > LGPL code. But we can depend on GPL or LGPL projects, perhaps even
> > maintained by people who are Apache committers wearing “different hats”,
> as
> > long as those dependencies are not mandatory.
> >
> > Julian
> >
> > [2] https://www.apache.org/legal/resolved.html#category-a
> >
> > > On Jun 24, 2021, at 11:05 AM, Hans Van Akelyen <
> > hans.van.akelyen@gmail.com> wrote:
> > >
> > > Hi Julian,
> > >
> > > But when creating a separate repository inside the Apache organisation
> I
> > > suppose all code needs to be Apache licensed?
> > > Or is it allowed to have (L)GPL repositories that merely act as
> optional
> > > dependencies for the main project?
> > >
> > > Cheers,
> > > Hans
> > >
> > > On Thu, 24 Jun 2021 at 20:01, Julian Hyde <jh...@gmail.com>
> > wrote:
> > >
> > >> I like the idea of a plugin repository too. Some plugins will be part
> of
> > >> Hop, and others will be external and merely referenced from a
> > “marketplace”.
> > >>
> > >> In case anyone is thinking of creating a ‘hop-plugins’ Git repo that’s
> > >> outside of Apache, let’s not go there. If the plugins are managed by
> the
> > >> Hop PMC (e.g. if Hop does the releases) then it is Apache code.
> > >>
> > >> A separate Git repo within Apache is fine, though. Many projects do
> > that.
> > >>
> > >> Now, regarding the original question, importing source code. Apache
> > >> requires that a "substantial contribution that was not developed
> within
> > the
> > >> ASF's source control system and on our public mailing lists” goes
> > through
> > >> IP clearance. This policy applies to incubator and top-level projects
> > >> alike, and is similar to the clearance process that we went though at
> > the
> > >> start of incubation. We need to follow that policy.
> > >>
> > >> No doubt you are wondering: what is “substantial”? Have a look at the
> > >> table in [1] to see what other projects have done.
> > >>
> > >>
> > >> Julian
> > >>
> > >> [1] https://incubator.apache.org/ip-clearance/
> > >>
> > >>> On Jun 24, 2021, at 8:37 AM, Brandon Jackson <us...@gmail.com>
> > >> wrote:
> > >>>
> > >>> I like Bart's idea of separating the repositories completely having
> an
> > >>> apache-hop-plugins repository along side the separate repository for
> > >> things
> > >>> that may not be compatible from a license standpoint.
> > >>>
> > >>> Since there are no autobuilds or packages sitting around of these
> other
> > >>> plugins, I think it makes it very difficult to discover and put the
> > >> plugins
> > >>> to use. Matt's idea of a marketplace of sorts would be nice.
> > >>> I liked the marketplace from prior times, even if it was built off an
> > XML
> > >>> submission. It was something usable.
> > >>>
> > >>> Brandon
> > >>>
> > >>>
> > >>> On Thu, Jun 24, 2021 at 2:45 AM Bart Maertens <bart.maertens@know.bi
> >
> > >> wrote:
> > >>>
> > >>>> Hi All,
> > >>>>
> > >>>>>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart
> Maertens
> > >>>> wrote
> > >>>>>>> the original version of that code so if he agrees to donate that
> > code
> > >>>> we're
> > >>>>>>> fine there.
> > >>>> There's probably no original code left, but absolutely fine by me
> > >>>>
> > >>>> Agreed on the process to let new plugins move through the
> hop-plugins
> > >>>> repository. We could use that repository as some sort of internal
> > >>>> incubating repository, where plugins have to mature before they are
> > >>>> accepted in the incubator-hop repository.
> > >>>>
> > >>>> We may need to think of a way to separate incubator-hop candidate
> > >> plugins
> > >>>> from plugins that can never make it into Apache Hop because of
> license
> > >> or
> > >>>> dependency incompatibilities (which is currently the case for all
> > >> plugins
> > >>>> in the repository). That is not an issue at the moment, but may get
> > >> harder
> > >>>> once the number of external plugins starts to grow.
> > >>>> One option could be to keep the current hop-plugins repository for
> > >>>> incompatible plugins and create an additional
> > >>>> apache-hop-plugins repository.
> > >>>>
> > >>>> Regards,
> > >>>> Bart
> > >>>>
> > >>>> On Wed, Jun 23, 2021 at 6:48 PM Matt Casters <
> matt.casters@neo4j.com
> > >>>> .invalid>
> > >>>> wrote:
> > >>>>
> > >>>>> Hello Devel-hoppers,
> > >>>>>
> > >>>>> Over the course of the last couple of months we've imported some
> > >>>> additional
> > >>>>> 3rd party source code.
> > >>>>> As was mentioned earlier during release voting we need to do a
> better
> > >> job
> > >>>>> of making sure that we have permission to donate said code to
> Apache.
> > >>>>> For everything related to my own code and the Neo4j plugins I can
> be
> > >>>> formal
> > >>>>> the following are donated:
> > >>>>> - Code we imported: kettle-beam, pentaho-pdi-dataset,
> > >>>> kettle-debug-plugin,
> > >>>>> kettle-needful-things, kettle-environment, kettle-splunk,
> > kettle-kafka
> > >>>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
> > >> wrote
> > >>>>> the original version of that code so if he agrees to donate that
> code
> > >>>> we're
> > >>>>> fine there.
> > >>>>>
> > >>>>> Obviously for as far as myself is concerned: everything I added to
> > the
> > >>>>> incubator-hop source code is also covered by my ICLA and a number
> of
> > >>>>> sources were mentioned in the incubation proposal.
> > >>>>> I'm just trying to be complete since apparently I've been a
> > proficient
> > >>>>> coder lately :-)
> > >>>>>
> > >>>>> Recently we removed some old SQL parser code (good riddance).
> > >>>>>
> > >>>>> Since everything in Hop is a plugin anyway I suggest that if we're
> > >> even a
> > >>>>> little bit uncertain about the origin of the plugin source code
> > (apart
> > >>>> from
> > >>>>> the original source code import/rewrite) that we simply move it out
> > >> into
> > >>>>> the external plugins <https://github.com/project-hop/hop-plugins>
> > >> until
> > >>>> we
> > >>>>> have clear permission of the original authors to use the code.
> Going
> > >>>>> forward I suggest we do the opposite. Let's add code to the
> external
> > >>>>> plugins project first and only then incorporate it into
> incubator-hop
> > >>>> when
> > >>>>> we have a more formal permission to donate the code to the ASF.
> > >>>>>
> > >>>>> I believe that the list of debatable plugins is this:
> > >>>>> - MQTT input/output (originally from Hitachi)
> > >>>>> - Google Sheets (informal permission received from JF Monteil, need
> > to
> > >>>>> formalize)
> > >>>>> - Google Analytics (originally from Hitachi)
> > >>>>> - Dropbox (permission needed from L. Coelho)
> > >>>>>
> > >>>>> The other new plugins were written from scratch and are fine I
> > believe.
> > >>>>>
> > >>>>> Personally I think it's just not worth it to even have these
> > >> discussions
> > >>>>> for a couple of obscure plugins so I would suggest moving this code
> > out
> > >>>> of
> > >>>>> our way until we have clear SGAs for these plugins.
> > >>>>>
> > >>>>> Let me know what your thoughts are! Perhaps if we have more
> external
> > >>>>> plugins it forces us to build a software marketplace at some point.
> > >>>>>
> > >>>>> Question for our mentors: in what forms do these agreements need to
> > >> come?
> > >>>>> Where are they stored and how are they communicated?
> > >>>>>
> > >>>>> Cheers,
> > >>>>> Matt
> > >>>>> --
> > >>>>> Neo4j Chief Solutions Architect
> > >>>>> *✉ *matt.casters@neo4j.com
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>
> --
> Neo4j Chief Solutions Architect
> *✉ *matt.casters@neo4j.com
>
Re: [DISCUSS] Imported source code
Posted by Matt Casters <ma...@neo4j.com.INVALID>.
I think in that case perhaps we can create an incubator-hop-external
repository or something like that which we don't release. We can build it
and perhaps even put it up somewhere.
The goal of that repository would be to evaluate licenses and
code-contributions. In case it's clear we're dealing with a conflicting
license the code can live in project-hop/hop-plugins or indeed anywhere
else.
In general I'd be quite happy with this state of affairs as it offers a
clear path into code contributions which we can document. Besides license
and copyright I would add other requirements like the need to have
integration tests, documentation and samples for the plugins.
I'll start a different discussion thread on the "marketplace" idea since I
think it'll become more important. There are already companies and
organizations porting their code and functionality over to the Apache Hop
API so we shouldn't wait too long to make a decision.
Cheers,
Matt
On Thu, Jun 24, 2021 at 10:22 PM Julian Hyde <jh...@gmail.com> wrote:
> Apache policy doesn’t really think in terms of Git repositories. An Apache
> PMC releases source code as a signed tar ball. (The PMC needs to be able to
> guarantee the provenance of that code, so in practice of course there is a
> source control system such as Git or Subversion.)
>
> If this plugin code is going to be released by the PMC, it must be under a
> "category A" license [2], which means Apache or another permissive license
> such as MIT or BSD. Such code can live in our source control system, once
> it has passed IP clearance.
>
> EPL is category B, which means that an Apache project can distribute
> binaries but not source code.
>
> GPL and LGPL are category X, which means that we cannot distribute
> binaries or source code. We can depend on a category X component (say a
> JDBC driver) via a plugin API as long as it isn’t the only possible
> implementation of that API.
>
> So, to answer your question: an Apache project cannot have GPL or LGPL
> repositories. There would be no point, because it cannot release GPL or
> LGPL code. But we can depend on GPL or LGPL projects, perhaps even
> maintained by people who are Apache committers wearing “different hats”, as
> long as those dependencies are not mandatory.
>
> Julian
>
> [2] https://www.apache.org/legal/resolved.html#category-a
>
> > On Jun 24, 2021, at 11:05 AM, Hans Van Akelyen <
> hans.van.akelyen@gmail.com> wrote:
> >
> > Hi Julian,
> >
> > But when creating a separate repository inside the Apache organisation I
> > suppose all code needs to be Apache licensed?
> > Or is it allowed to have (L)GPL repositories that merely act as optional
> > dependencies for the main project?
> >
> > Cheers,
> > Hans
> >
> > On Thu, 24 Jun 2021 at 20:01, Julian Hyde <jh...@gmail.com>
> wrote:
> >
> >> I like the idea of a plugin repository too. Some plugins will be part of
> >> Hop, and others will be external and merely referenced from a
> “marketplace”.
> >>
> >> In case anyone is thinking of creating a ‘hop-plugins’ Git repo that’s
> >> outside of Apache, let’s not go there. If the plugins are managed by the
> >> Hop PMC (e.g. if Hop does the releases) then it is Apache code.
> >>
> >> A separate Git repo within Apache is fine, though. Many projects do
> that.
> >>
> >> Now, regarding the original question, importing source code. Apache
> >> requires that a "substantial contribution that was not developed within
> the
> >> ASF's source control system and on our public mailing lists” goes
> through
> >> IP clearance. This policy applies to incubator and top-level projects
> >> alike, and is similar to the clearance process that we went though at
> the
> >> start of incubation. We need to follow that policy.
> >>
> >> No doubt you are wondering: what is “substantial”? Have a look at the
> >> table in [1] to see what other projects have done.
> >>
> >>
> >> Julian
> >>
> >> [1] https://incubator.apache.org/ip-clearance/
> >>
> >>> On Jun 24, 2021, at 8:37 AM, Brandon Jackson <us...@gmail.com>
> >> wrote:
> >>>
> >>> I like Bart's idea of separating the repositories completely having an
> >>> apache-hop-plugins repository along side the separate repository for
> >> things
> >>> that may not be compatible from a license standpoint.
> >>>
> >>> Since there are no autobuilds or packages sitting around of these other
> >>> plugins, I think it makes it very difficult to discover and put the
> >> plugins
> >>> to use. Matt's idea of a marketplace of sorts would be nice.
> >>> I liked the marketplace from prior times, even if it was built off an
> XML
> >>> submission. It was something usable.
> >>>
> >>> Brandon
> >>>
> >>>
> >>> On Thu, Jun 24, 2021 at 2:45 AM Bart Maertens <ba...@know.bi>
> >> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>>>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
> >>>> wrote
> >>>>>>> the original version of that code so if he agrees to donate that
> code
> >>>> we're
> >>>>>>> fine there.
> >>>> There's probably no original code left, but absolutely fine by me
> >>>>
> >>>> Agreed on the process to let new plugins move through the hop-plugins
> >>>> repository. We could use that repository as some sort of internal
> >>>> incubating repository, where plugins have to mature before they are
> >>>> accepted in the incubator-hop repository.
> >>>>
> >>>> We may need to think of a way to separate incubator-hop candidate
> >> plugins
> >>>> from plugins that can never make it into Apache Hop because of license
> >> or
> >>>> dependency incompatibilities (which is currently the case for all
> >> plugins
> >>>> in the repository). That is not an issue at the moment, but may get
> >> harder
> >>>> once the number of external plugins starts to grow.
> >>>> One option could be to keep the current hop-plugins repository for
> >>>> incompatible plugins and create an additional
> >>>> apache-hop-plugins repository.
> >>>>
> >>>> Regards,
> >>>> Bart
> >>>>
> >>>> On Wed, Jun 23, 2021 at 6:48 PM Matt Casters <matt.casters@neo4j.com
> >>>> .invalid>
> >>>> wrote:
> >>>>
> >>>>> Hello Devel-hoppers,
> >>>>>
> >>>>> Over the course of the last couple of months we've imported some
> >>>> additional
> >>>>> 3rd party source code.
> >>>>> As was mentioned earlier during release voting we need to do a better
> >> job
> >>>>> of making sure that we have permission to donate said code to Apache.
> >>>>> For everything related to my own code and the Neo4j plugins I can be
> >>>> formal
> >>>>> the following are donated:
> >>>>> - Code we imported: kettle-beam, pentaho-pdi-dataset,
> >>>> kettle-debug-plugin,
> >>>>> kettle-needful-things, kettle-environment, kettle-splunk,
> kettle-kafka
> >>>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
> >> wrote
> >>>>> the original version of that code so if he agrees to donate that code
> >>>> we're
> >>>>> fine there.
> >>>>>
> >>>>> Obviously for as far as myself is concerned: everything I added to
> the
> >>>>> incubator-hop source code is also covered by my ICLA and a number of
> >>>>> sources were mentioned in the incubation proposal.
> >>>>> I'm just trying to be complete since apparently I've been a
> proficient
> >>>>> coder lately :-)
> >>>>>
> >>>>> Recently we removed some old SQL parser code (good riddance).
> >>>>>
> >>>>> Since everything in Hop is a plugin anyway I suggest that if we're
> >> even a
> >>>>> little bit uncertain about the origin of the plugin source code
> (apart
> >>>> from
> >>>>> the original source code import/rewrite) that we simply move it out
> >> into
> >>>>> the external plugins <https://github.com/project-hop/hop-plugins>
> >> until
> >>>> we
> >>>>> have clear permission of the original authors to use the code. Going
> >>>>> forward I suggest we do the opposite. Let's add code to the external
> >>>>> plugins project first and only then incorporate it into incubator-hop
> >>>> when
> >>>>> we have a more formal permission to donate the code to the ASF.
> >>>>>
> >>>>> I believe that the list of debatable plugins is this:
> >>>>> - MQTT input/output (originally from Hitachi)
> >>>>> - Google Sheets (informal permission received from JF Monteil, need
> to
> >>>>> formalize)
> >>>>> - Google Analytics (originally from Hitachi)
> >>>>> - Dropbox (permission needed from L. Coelho)
> >>>>>
> >>>>> The other new plugins were written from scratch and are fine I
> believe.
> >>>>>
> >>>>> Personally I think it's just not worth it to even have these
> >> discussions
> >>>>> for a couple of obscure plugins so I would suggest moving this code
> out
> >>>> of
> >>>>> our way until we have clear SGAs for these plugins.
> >>>>>
> >>>>> Let me know what your thoughts are! Perhaps if we have more external
> >>>>> plugins it forces us to build a software marketplace at some point.
> >>>>>
> >>>>> Question for our mentors: in what forms do these agreements need to
> >> come?
> >>>>> Where are they stored and how are they communicated?
> >>>>>
> >>>>> Cheers,
> >>>>> Matt
> >>>>> --
> >>>>> Neo4j Chief Solutions Architect
> >>>>> *✉ *matt.casters@neo4j.com
> >>>>>
> >>>>
> >>
> >>
>
>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
Re: [DISCUSS] Imported source code
Posted by Julian Hyde <jh...@gmail.com>.
Apache policy doesn’t really think in terms of Git repositories. An Apache PMC releases source code as a signed tar ball. (The PMC needs to be able to guarantee the provenance of that code, so in practice of course there is a source control system such as Git or Subversion.)
If this plugin code is going to be released by the PMC, it must be under a "category A" license [2], which means Apache or another permissive license such as MIT or BSD. Such code can live in our source control system, once it has passed IP clearance.
EPL is category B, which means that an Apache project can distribute binaries but not source code.
GPL and LGPL are category X, which means that we cannot distribute binaries or source code. We can depend on a category X component (say a JDBC driver) via a plugin API as long as it isn’t the only possible implementation of that API.
So, to answer your question: an Apache project cannot have GPL or LGPL repositories. There would be no point, because it cannot release GPL or LGPL code. But we can depend on GPL or LGPL projects, perhaps even maintained by people who are Apache committers wearing “different hats”, as long as those dependencies are not mandatory.
Julian
[2] https://www.apache.org/legal/resolved.html#category-a
> On Jun 24, 2021, at 11:05 AM, Hans Van Akelyen <ha...@gmail.com> wrote:
>
> Hi Julian,
>
> But when creating a separate repository inside the Apache organisation I
> suppose all code needs to be Apache licensed?
> Or is it allowed to have (L)GPL repositories that merely act as optional
> dependencies for the main project?
>
> Cheers,
> Hans
>
> On Thu, 24 Jun 2021 at 20:01, Julian Hyde <jh...@gmail.com> wrote:
>
>> I like the idea of a plugin repository too. Some plugins will be part of
>> Hop, and others will be external and merely referenced from a “marketplace”.
>>
>> In case anyone is thinking of creating a ‘hop-plugins’ Git repo that’s
>> outside of Apache, let’s not go there. If the plugins are managed by the
>> Hop PMC (e.g. if Hop does the releases) then it is Apache code.
>>
>> A separate Git repo within Apache is fine, though. Many projects do that.
>>
>> Now, regarding the original question, importing source code. Apache
>> requires that a "substantial contribution that was not developed within the
>> ASF's source control system and on our public mailing lists” goes through
>> IP clearance. This policy applies to incubator and top-level projects
>> alike, and is similar to the clearance process that we went though at the
>> start of incubation. We need to follow that policy.
>>
>> No doubt you are wondering: what is “substantial”? Have a look at the
>> table in [1] to see what other projects have done.
>>
>>
>> Julian
>>
>> [1] https://incubator.apache.org/ip-clearance/
>>
>>> On Jun 24, 2021, at 8:37 AM, Brandon Jackson <us...@gmail.com>
>> wrote:
>>>
>>> I like Bart's idea of separating the repositories completely having an
>>> apache-hop-plugins repository along side the separate repository for
>> things
>>> that may not be compatible from a license standpoint.
>>>
>>> Since there are no autobuilds or packages sitting around of these other
>>> plugins, I think it makes it very difficult to discover and put the
>> plugins
>>> to use. Matt's idea of a marketplace of sorts would be nice.
>>> I liked the marketplace from prior times, even if it was built off an XML
>>> submission. It was something usable.
>>>
>>> Brandon
>>>
>>>
>>> On Thu, Jun 24, 2021 at 2:45 AM Bart Maertens <ba...@know.bi>
>> wrote:
>>>
>>>> Hi All,
>>>>
>>>>>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
>>>> wrote
>>>>>>> the original version of that code so if he agrees to donate that code
>>>> we're
>>>>>>> fine there.
>>>> There's probably no original code left, but absolutely fine by me
>>>>
>>>> Agreed on the process to let new plugins move through the hop-plugins
>>>> repository. We could use that repository as some sort of internal
>>>> incubating repository, where plugins have to mature before they are
>>>> accepted in the incubator-hop repository.
>>>>
>>>> We may need to think of a way to separate incubator-hop candidate
>> plugins
>>>> from plugins that can never make it into Apache Hop because of license
>> or
>>>> dependency incompatibilities (which is currently the case for all
>> plugins
>>>> in the repository). That is not an issue at the moment, but may get
>> harder
>>>> once the number of external plugins starts to grow.
>>>> One option could be to keep the current hop-plugins repository for
>>>> incompatible plugins and create an additional
>>>> apache-hop-plugins repository.
>>>>
>>>> Regards,
>>>> Bart
>>>>
>>>> On Wed, Jun 23, 2021 at 6:48 PM Matt Casters <matt.casters@neo4j.com
>>>> .invalid>
>>>> wrote:
>>>>
>>>>> Hello Devel-hoppers,
>>>>>
>>>>> Over the course of the last couple of months we've imported some
>>>> additional
>>>>> 3rd party source code.
>>>>> As was mentioned earlier during release voting we need to do a better
>> job
>>>>> of making sure that we have permission to donate said code to Apache.
>>>>> For everything related to my own code and the Neo4j plugins I can be
>>>> formal
>>>>> the following are donated:
>>>>> - Code we imported: kettle-beam, pentaho-pdi-dataset,
>>>> kettle-debug-plugin,
>>>>> kettle-needful-things, kettle-environment, kettle-splunk, kettle-kafka
>>>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
>> wrote
>>>>> the original version of that code so if he agrees to donate that code
>>>> we're
>>>>> fine there.
>>>>>
>>>>> Obviously for as far as myself is concerned: everything I added to the
>>>>> incubator-hop source code is also covered by my ICLA and a number of
>>>>> sources were mentioned in the incubation proposal.
>>>>> I'm just trying to be complete since apparently I've been a proficient
>>>>> coder lately :-)
>>>>>
>>>>> Recently we removed some old SQL parser code (good riddance).
>>>>>
>>>>> Since everything in Hop is a plugin anyway I suggest that if we're
>> even a
>>>>> little bit uncertain about the origin of the plugin source code (apart
>>>> from
>>>>> the original source code import/rewrite) that we simply move it out
>> into
>>>>> the external plugins <https://github.com/project-hop/hop-plugins>
>> until
>>>> we
>>>>> have clear permission of the original authors to use the code. Going
>>>>> forward I suggest we do the opposite. Let's add code to the external
>>>>> plugins project first and only then incorporate it into incubator-hop
>>>> when
>>>>> we have a more formal permission to donate the code to the ASF.
>>>>>
>>>>> I believe that the list of debatable plugins is this:
>>>>> - MQTT input/output (originally from Hitachi)
>>>>> - Google Sheets (informal permission received from JF Monteil, need to
>>>>> formalize)
>>>>> - Google Analytics (originally from Hitachi)
>>>>> - Dropbox (permission needed from L. Coelho)
>>>>>
>>>>> The other new plugins were written from scratch and are fine I believe.
>>>>>
>>>>> Personally I think it's just not worth it to even have these
>> discussions
>>>>> for a couple of obscure plugins so I would suggest moving this code out
>>>> of
>>>>> our way until we have clear SGAs for these plugins.
>>>>>
>>>>> Let me know what your thoughts are! Perhaps if we have more external
>>>>> plugins it forces us to build a software marketplace at some point.
>>>>>
>>>>> Question for our mentors: in what forms do these agreements need to
>> come?
>>>>> Where are they stored and how are they communicated?
>>>>>
>>>>> Cheers,
>>>>> Matt
>>>>> --
>>>>> Neo4j Chief Solutions Architect
>>>>> *✉ *matt.casters@neo4j.com
>>>>>
>>>>
>>
>>
Re: [DISCUSS] Imported source code
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi Julian,
But when creating a separate repository inside the Apache organisation I
suppose all code needs to be Apache licensed?
Or is it allowed to have (L)GPL repositories that merely act as optional
dependencies for the main project?
Cheers,
Hans
On Thu, 24 Jun 2021 at 20:01, Julian Hyde <jh...@gmail.com> wrote:
> I like the idea of a plugin repository too. Some plugins will be part of
> Hop, and others will be external and merely referenced from a “marketplace”.
>
> In case anyone is thinking of creating a ‘hop-plugins’ Git repo that’s
> outside of Apache, let’s not go there. If the plugins are managed by the
> Hop PMC (e.g. if Hop does the releases) then it is Apache code.
>
> A separate Git repo within Apache is fine, though. Many projects do that.
>
> Now, regarding the original question, importing source code. Apache
> requires that a "substantial contribution that was not developed within the
> ASF's source control system and on our public mailing lists” goes through
> IP clearance. This policy applies to incubator and top-level projects
> alike, and is similar to the clearance process that we went though at the
> start of incubation. We need to follow that policy.
>
> No doubt you are wondering: what is “substantial”? Have a look at the
> table in [1] to see what other projects have done.
>
>
> Julian
>
> [1] https://incubator.apache.org/ip-clearance/
>
> > On Jun 24, 2021, at 8:37 AM, Brandon Jackson <us...@gmail.com>
> wrote:
> >
> > I like Bart's idea of separating the repositories completely having an
> > apache-hop-plugins repository along side the separate repository for
> things
> > that may not be compatible from a license standpoint.
> >
> > Since there are no autobuilds or packages sitting around of these other
> > plugins, I think it makes it very difficult to discover and put the
> plugins
> > to use. Matt's idea of a marketplace of sorts would be nice.
> > I liked the marketplace from prior times, even if it was built off an XML
> > submission. It was something usable.
> >
> > Brandon
> >
> >
> > On Thu, Jun 24, 2021 at 2:45 AM Bart Maertens <ba...@know.bi>
> wrote:
> >
> >> Hi All,
> >>
> >>>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
> >> wrote
> >>>>> the original version of that code so if he agrees to donate that code
> >> we're
> >>>>> fine there.
> >> There's probably no original code left, but absolutely fine by me
> >>
> >> Agreed on the process to let new plugins move through the hop-plugins
> >> repository. We could use that repository as some sort of internal
> >> incubating repository, where plugins have to mature before they are
> >> accepted in the incubator-hop repository.
> >>
> >> We may need to think of a way to separate incubator-hop candidate
> plugins
> >> from plugins that can never make it into Apache Hop because of license
> or
> >> dependency incompatibilities (which is currently the case for all
> plugins
> >> in the repository). That is not an issue at the moment, but may get
> harder
> >> once the number of external plugins starts to grow.
> >> One option could be to keep the current hop-plugins repository for
> >> incompatible plugins and create an additional
> >> apache-hop-plugins repository.
> >>
> >> Regards,
> >> Bart
> >>
> >> On Wed, Jun 23, 2021 at 6:48 PM Matt Casters <matt.casters@neo4j.com
> >> .invalid>
> >> wrote:
> >>
> >>> Hello Devel-hoppers,
> >>>
> >>> Over the course of the last couple of months we've imported some
> >> additional
> >>> 3rd party source code.
> >>> As was mentioned earlier during release voting we need to do a better
> job
> >>> of making sure that we have permission to donate said code to Apache.
> >>> For everything related to my own code and the Neo4j plugins I can be
> >> formal
> >>> the following are donated:
> >>> - Code we imported: kettle-beam, pentaho-pdi-dataset,
> >> kettle-debug-plugin,
> >>> kettle-needful-things, kettle-environment, kettle-splunk, kettle-kafka
> >>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
> wrote
> >>> the original version of that code so if he agrees to donate that code
> >> we're
> >>> fine there.
> >>>
> >>> Obviously for as far as myself is concerned: everything I added to the
> >>> incubator-hop source code is also covered by my ICLA and a number of
> >>> sources were mentioned in the incubation proposal.
> >>> I'm just trying to be complete since apparently I've been a proficient
> >>> coder lately :-)
> >>>
> >>> Recently we removed some old SQL parser code (good riddance).
> >>>
> >>> Since everything in Hop is a plugin anyway I suggest that if we're
> even a
> >>> little bit uncertain about the origin of the plugin source code (apart
> >> from
> >>> the original source code import/rewrite) that we simply move it out
> into
> >>> the external plugins <https://github.com/project-hop/hop-plugins>
> until
> >> we
> >>> have clear permission of the original authors to use the code. Going
> >>> forward I suggest we do the opposite. Let's add code to the external
> >>> plugins project first and only then incorporate it into incubator-hop
> >> when
> >>> we have a more formal permission to donate the code to the ASF.
> >>>
> >>> I believe that the list of debatable plugins is this:
> >>> - MQTT input/output (originally from Hitachi)
> >>> - Google Sheets (informal permission received from JF Monteil, need to
> >>> formalize)
> >>> - Google Analytics (originally from Hitachi)
> >>> - Dropbox (permission needed from L. Coelho)
> >>>
> >>> The other new plugins were written from scratch and are fine I believe.
> >>>
> >>> Personally I think it's just not worth it to even have these
> discussions
> >>> for a couple of obscure plugins so I would suggest moving this code out
> >> of
> >>> our way until we have clear SGAs for these plugins.
> >>>
> >>> Let me know what your thoughts are! Perhaps if we have more external
> >>> plugins it forces us to build a software marketplace at some point.
> >>>
> >>> Question for our mentors: in what forms do these agreements need to
> come?
> >>> Where are they stored and how are they communicated?
> >>>
> >>> Cheers,
> >>> Matt
> >>> --
> >>> Neo4j Chief Solutions Architect
> >>> *✉ *matt.casters@neo4j.com
> >>>
> >>
>
>
Re: [DISCUSS] Imported source code
Posted by Julian Hyde <jh...@gmail.com>.
I like the idea of a plugin repository too. Some plugins will be part of Hop, and others will be external and merely referenced from a “marketplace”.
In case anyone is thinking of creating a ‘hop-plugins’ Git repo that’s outside of Apache, let’s not go there. If the plugins are managed by the Hop PMC (e.g. if Hop does the releases) then it is Apache code.
A separate Git repo within Apache is fine, though. Many projects do that.
Now, regarding the original question, importing source code. Apache requires that a "substantial contribution that was not developed within the ASF's source control system and on our public mailing lists” goes through IP clearance. This policy applies to incubator and top-level projects alike, and is similar to the clearance process that we went though at the start of incubation. We need to follow that policy.
No doubt you are wondering: what is “substantial”? Have a look at the table in [1] to see what other projects have done.
Julian
[1] https://incubator.apache.org/ip-clearance/
> On Jun 24, 2021, at 8:37 AM, Brandon Jackson <us...@gmail.com> wrote:
>
> I like Bart's idea of separating the repositories completely having an
> apache-hop-plugins repository along side the separate repository for things
> that may not be compatible from a license standpoint.
>
> Since there are no autobuilds or packages sitting around of these other
> plugins, I think it makes it very difficult to discover and put the plugins
> to use. Matt's idea of a marketplace of sorts would be nice.
> I liked the marketplace from prior times, even if it was built off an XML
> submission. It was something usable.
>
> Brandon
>
>
> On Thu, Jun 24, 2021 at 2:45 AM Bart Maertens <ba...@know.bi> wrote:
>
>> Hi All,
>>
>>>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
>> wrote
>>>>> the original version of that code so if he agrees to donate that code
>> we're
>>>>> fine there.
>> There's probably no original code left, but absolutely fine by me
>>
>> Agreed on the process to let new plugins move through the hop-plugins
>> repository. We could use that repository as some sort of internal
>> incubating repository, where plugins have to mature before they are
>> accepted in the incubator-hop repository.
>>
>> We may need to think of a way to separate incubator-hop candidate plugins
>> from plugins that can never make it into Apache Hop because of license or
>> dependency incompatibilities (which is currently the case for all plugins
>> in the repository). That is not an issue at the moment, but may get harder
>> once the number of external plugins starts to grow.
>> One option could be to keep the current hop-plugins repository for
>> incompatible plugins and create an additional
>> apache-hop-plugins repository.
>>
>> Regards,
>> Bart
>>
>> On Wed, Jun 23, 2021 at 6:48 PM Matt Casters <matt.casters@neo4j.com
>> .invalid>
>> wrote:
>>
>>> Hello Devel-hoppers,
>>>
>>> Over the course of the last couple of months we've imported some
>> additional
>>> 3rd party source code.
>>> As was mentioned earlier during release voting we need to do a better job
>>> of making sure that we have permission to donate said code to Apache.
>>> For everything related to my own code and the Neo4j plugins I can be
>> formal
>>> the following are donated:
>>> - Code we imported: kettle-beam, pentaho-pdi-dataset,
>> kettle-debug-plugin,
>>> kettle-needful-things, kettle-environment, kettle-splunk, kettle-kafka
>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens wrote
>>> the original version of that code so if he agrees to donate that code
>> we're
>>> fine there.
>>>
>>> Obviously for as far as myself is concerned: everything I added to the
>>> incubator-hop source code is also covered by my ICLA and a number of
>>> sources were mentioned in the incubation proposal.
>>> I'm just trying to be complete since apparently I've been a proficient
>>> coder lately :-)
>>>
>>> Recently we removed some old SQL parser code (good riddance).
>>>
>>> Since everything in Hop is a plugin anyway I suggest that if we're even a
>>> little bit uncertain about the origin of the plugin source code (apart
>> from
>>> the original source code import/rewrite) that we simply move it out into
>>> the external plugins <https://github.com/project-hop/hop-plugins> until
>> we
>>> have clear permission of the original authors to use the code. Going
>>> forward I suggest we do the opposite. Let's add code to the external
>>> plugins project first and only then incorporate it into incubator-hop
>> when
>>> we have a more formal permission to donate the code to the ASF.
>>>
>>> I believe that the list of debatable plugins is this:
>>> - MQTT input/output (originally from Hitachi)
>>> - Google Sheets (informal permission received from JF Monteil, need to
>>> formalize)
>>> - Google Analytics (originally from Hitachi)
>>> - Dropbox (permission needed from L. Coelho)
>>>
>>> The other new plugins were written from scratch and are fine I believe.
>>>
>>> Personally I think it's just not worth it to even have these discussions
>>> for a couple of obscure plugins so I would suggest moving this code out
>> of
>>> our way until we have clear SGAs for these plugins.
>>>
>>> Let me know what your thoughts are! Perhaps if we have more external
>>> plugins it forces us to build a software marketplace at some point.
>>>
>>> Question for our mentors: in what forms do these agreements need to come?
>>> Where are they stored and how are they communicated?
>>>
>>> Cheers,
>>> Matt
>>> --
>>> Neo4j Chief Solutions Architect
>>> *✉ *matt.casters@neo4j.com
>>>
>>
Re: [DISCUSS] Imported source code
Posted by Brandon Jackson <us...@gmail.com>.
I like Bart's idea of separating the repositories completely having an
apache-hop-plugins repository along side the separate repository for things
that may not be compatible from a license standpoint.
Since there are no autobuilds or packages sitting around of these other
plugins, I think it makes it very difficult to discover and put the plugins
to use. Matt's idea of a marketplace of sorts would be nice.
I liked the marketplace from prior times, even if it was built off an XML
submission. It was something usable.
Brandon
On Thu, Jun 24, 2021 at 2:45 AM Bart Maertens <ba...@know.bi> wrote:
> Hi All,
>
> >>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
> wrote
> >>> the original version of that code so if he agrees to donate that code
> we're
> >>> fine there.
> There's probably no original code left, but absolutely fine by me
>
> Agreed on the process to let new plugins move through the hop-plugins
> repository. We could use that repository as some sort of internal
> incubating repository, where plugins have to mature before they are
> accepted in the incubator-hop repository.
>
> We may need to think of a way to separate incubator-hop candidate plugins
> from plugins that can never make it into Apache Hop because of license or
> dependency incompatibilities (which is currently the case for all plugins
> in the repository). That is not an issue at the moment, but may get harder
> once the number of external plugins starts to grow.
> One option could be to keep the current hop-plugins repository for
> incompatible plugins and create an additional
> apache-hop-plugins repository.
>
> Regards,
> Bart
>
> On Wed, Jun 23, 2021 at 6:48 PM Matt Casters <matt.casters@neo4j.com
> .invalid>
> wrote:
>
> > Hello Devel-hoppers,
> >
> > Over the course of the last couple of months we've imported some
> additional
> > 3rd party source code.
> > As was mentioned earlier during release voting we need to do a better job
> > of making sure that we have permission to donate said code to Apache.
> > For everything related to my own code and the Neo4j plugins I can be
> formal
> > the following are donated:
> > - Code we imported: kettle-beam, pentaho-pdi-dataset,
> kettle-debug-plugin,
> > kettle-needful-things, kettle-environment, kettle-splunk, kettle-kafka
> > - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens wrote
> > the original version of that code so if he agrees to donate that code
> we're
> > fine there.
> >
> > Obviously for as far as myself is concerned: everything I added to the
> > incubator-hop source code is also covered by my ICLA and a number of
> > sources were mentioned in the incubation proposal.
> > I'm just trying to be complete since apparently I've been a proficient
> > coder lately :-)
> >
> > Recently we removed some old SQL parser code (good riddance).
> >
> > Since everything in Hop is a plugin anyway I suggest that if we're even a
> > little bit uncertain about the origin of the plugin source code (apart
> from
> > the original source code import/rewrite) that we simply move it out into
> > the external plugins <https://github.com/project-hop/hop-plugins> until
> we
> > have clear permission of the original authors to use the code. Going
> > forward I suggest we do the opposite. Let's add code to the external
> > plugins project first and only then incorporate it into incubator-hop
> when
> > we have a more formal permission to donate the code to the ASF.
> >
> > I believe that the list of debatable plugins is this:
> > - MQTT input/output (originally from Hitachi)
> > - Google Sheets (informal permission received from JF Monteil, need to
> > formalize)
> > - Google Analytics (originally from Hitachi)
> > - Dropbox (permission needed from L. Coelho)
> >
> > The other new plugins were written from scratch and are fine I believe.
> >
> > Personally I think it's just not worth it to even have these discussions
> > for a couple of obscure plugins so I would suggest moving this code out
> of
> > our way until we have clear SGAs for these plugins.
> >
> > Let me know what your thoughts are! Perhaps if we have more external
> > plugins it forces us to build a software marketplace at some point.
> >
> > Question for our mentors: in what forms do these agreements need to come?
> > Where are they stored and how are they communicated?
> >
> > Cheers,
> > Matt
> > --
> > Neo4j Chief Solutions Architect
> > *✉ *matt.casters@neo4j.com
> >
>
Re: [DISCUSS] Imported source code
Posted by Bart Maertens <ba...@know.bi>.
Hi All,
>>> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens
wrote
>>> the original version of that code so if he agrees to donate that code
we're
>>> fine there.
There's probably no original code left, but absolutely fine by me
Agreed on the process to let new plugins move through the hop-plugins
repository. We could use that repository as some sort of internal
incubating repository, where plugins have to mature before they are
accepted in the incubator-hop repository.
We may need to think of a way to separate incubator-hop candidate plugins
from plugins that can never make it into Apache Hop because of license or
dependency incompatibilities (which is currently the case for all plugins
in the repository). That is not an issue at the moment, but may get harder
once the number of external plugins starts to grow.
One option could be to keep the current hop-plugins repository for
incompatible plugins and create an additional
apache-hop-plugins repository.
Regards,
Bart
On Wed, Jun 23, 2021 at 6:48 PM Matt Casters <ma...@neo4j.com.invalid>
wrote:
> Hello Devel-hoppers,
>
> Over the course of the last couple of months we've imported some additional
> 3rd party source code.
> As was mentioned earlier during release voting we need to do a better job
> of making sure that we have permission to donate said code to Apache.
> For everything related to my own code and the Neo4j plugins I can be formal
> the following are donated:
> - Code we imported: kettle-beam, pentaho-pdi-dataset, kettle-debug-plugin,
> kettle-needful-things, kettle-environment, kettle-splunk, kettle-kafka
> - Neo4j: The Neo4j Output plugins. I do believe that Bart Maertens wrote
> the original version of that code so if he agrees to donate that code we're
> fine there.
>
> Obviously for as far as myself is concerned: everything I added to the
> incubator-hop source code is also covered by my ICLA and a number of
> sources were mentioned in the incubation proposal.
> I'm just trying to be complete since apparently I've been a proficient
> coder lately :-)
>
> Recently we removed some old SQL parser code (good riddance).
>
> Since everything in Hop is a plugin anyway I suggest that if we're even a
> little bit uncertain about the origin of the plugin source code (apart from
> the original source code import/rewrite) that we simply move it out into
> the external plugins <https://github.com/project-hop/hop-plugins> until we
> have clear permission of the original authors to use the code. Going
> forward I suggest we do the opposite. Let's add code to the external
> plugins project first and only then incorporate it into incubator-hop when
> we have a more formal permission to donate the code to the ASF.
>
> I believe that the list of debatable plugins is this:
> - MQTT input/output (originally from Hitachi)
> - Google Sheets (informal permission received from JF Monteil, need to
> formalize)
> - Google Analytics (originally from Hitachi)
> - Dropbox (permission needed from L. Coelho)
>
> The other new plugins were written from scratch and are fine I believe.
>
> Personally I think it's just not worth it to even have these discussions
> for a couple of obscure plugins so I would suggest moving this code out of
> our way until we have clear SGAs for these plugins.
>
> Let me know what your thoughts are! Perhaps if we have more external
> plugins it forces us to build a software marketplace at some point.
>
> Question for our mentors: in what forms do these agreements need to come?
> Where are they stored and how are they communicated?
>
> Cheers,
> Matt
> --
> Neo4j Chief Solutions Architect
> *✉ *matt.casters@neo4j.com
>