You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sean Owen <so...@cloudera.com> on 2016/09/07 08:29:05 UTC
Removing published kinesis, ganglia artifacts due to license issues?
It's worth calling attention to:
https://issues.apache.org/jira/browse/SPARK-17418
https://issues.apache.org/jira/browse/SPARK-17422
It looks like we need to at least not publish the kinesis *assembly*
Maven artifact because it contains Amazon Software Licensed-code
directly.
However there's a reasonably strong reason to believe that we'd have
to remove the non-assembly Kinesis artifact too, as well as the
Ganglia one. This doesn't mean it goes away from the project, just
means it would no longer be published as a Maven artifact. (These have
never been bundled in the main Spark artifacts.)
I wanted to give a heads up to see if anyone a) believes this
conclusion is wrong or b) wants to take it up with legal@? I'm
inclined to believe we have to remove them given the interpretation
Luciano has put forth.
Sean
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Mridul Muralidharan <mr...@gmail.com>.
I agree, we should not be publishing both of them.
Thanks for bringing this up !
Regards,
Mridul
On Wed, Sep 7, 2016 at 1:29 AM, Sean Owen <so...@cloudera.com> wrote:
> It's worth calling attention to:
>
> https://issues.apache.org/jira/browse/SPARK-17418
> https://issues.apache.org/jira/browse/SPARK-17422
>
> It looks like we need to at least not publish the kinesis *assembly*
> Maven artifact because it contains Amazon Software Licensed-code
> directly.
>
> However there's a reasonably strong reason to believe that we'd have
> to remove the non-assembly Kinesis artifact too, as well as the
> Ganglia one. This doesn't mean it goes away from the project, just
> means it would no longer be published as a Maven artifact. (These have
> never been bundled in the main Spark artifacts.)
>
> I wanted to give a heads up to see if anyone a) believes this
> conclusion is wrong or b) wants to take it up with legal@? I'm
> inclined to believe we have to remove them given the interpretation
> Luciano has put forth.
>
> Sean
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Luciano Resende <lu...@gmail.com>.
On Wed, Sep 7, 2016 at 11:57 AM, Matei Zaharia <ma...@gmail.com>
wrote:
> I think you should ask legal about how to have some Maven artifacts for
> these. Both Ganglia and Kinesis are very widely used, so it's weird to ask
> users to build them from source. Maybe the Maven artifacts can be marked as
> being under a different license?
>
>
As long as they are not part of an "Apache licensed" distribution. Note
that Ganglia seems to have changed license to BSD and we might be able to
better support that.
> In the initial discussion for LEGAL-198, we were told the following:
>
> "If the component that uses this dependency is not required for the rest
> of Spark to function then you can have a subproject to build the component.
> See http://www.apache.org/legal/resolved.html#optional. This means you
> will have to provide instructions for users to enable the optional
> component (which IMO should provide pointers to the licensing)."
>
> It's not clear whether "enable the optional component" means "every user
> must build it from source", or whether we could tell users "here's a Maven
> coordinate you can add to your project if you're okay with the licensing".
>
I think the key here is "optional", while the Kinesis is optional for Spark
(which makes it ok to have it in Spark) it is not optional for Kinesis
extension, which thenm IMHO, does not allow us to publish the Kinesis
artifact either.
But let's wait on the response from Legal before we actually implement a
solution.
>
> Matei
>
> > On Sep 7, 2016, at 11:35 AM, Sean Owen <so...@cloudera.com> wrote:
> >
> > (Credit to Luciano for pointing it out)
> >
> > Yes it's clear why the assembly can't be published but I had the same
> > question about the non-assembly Kinesis (and ganglia) artifact,
> > because the published artifact has no code from Kinesis.
> >
> > See the related discussion at
> > https://issues.apache.org/jira/browse/LEGAL-198 ; the point I took
> > from there is that the Spark Kinesis artifact is optional with respect
> > to Spark, but still something published by Spark, and it requires the
> > Amazon-licensed code non-optionally.
> >
> > I'll just ask that question to confirm or deny.
> >
> > (It also has some background on why the Amazon License is considered
> > "Category X" in ASF policy due to field of use restrictions. I myself
> > take that as read rather than know the details of that decision.)
> >
> > On Wed, Sep 7, 2016 at 6:44 PM, Cody Koeninger <co...@koeninger.org>
> wrote:
> >> I don't see a reason to remove the non-assembly artifact, why would
> >> you? You're not distributing copies of Amazon licensed code, and the
> >> Amazon license goes out of its way not to over-reach regarding
> >> derivative works.
> >>
> >> This seems pretty clearly to fall in the spirit of
> >>
> >> http://www.apache.org/legal/resolved.html#optional
> >>
> >> I certainly think the majority of Spark users will still want to use
> >> Spark without adding Kinesis
> >>
> >> On Wed, Sep 7, 2016 at 3:29 AM, Sean Owen <so...@cloudera.com> wrote:
> >>> It's worth calling attention to:
> >>>
> >>> https://issues.apache.org/jira/browse/SPARK-17418
> >>> https://issues.apache.org/jira/browse/SPARK-17422
> >>>
> >>> It looks like we need to at least not publish the kinesis *assembly*
> >>> Maven artifact because it contains Amazon Software Licensed-code
> >>> directly.
> >>>
> >>> However there's a reasonably strong reason to believe that we'd have
> >>> to remove the non-assembly Kinesis artifact too, as well as the
> >>> Ganglia one. This doesn't mean it goes away from the project, just
> >>> means it would no longer be published as a Maven artifact. (These have
> >>> never been bundled in the main Spark artifacts.)
> >>>
> >>> I wanted to give a heads up to see if anyone a) believes this
> >>> conclusion is wrong or b) wants to take it up with legal@? I'm
> >>> inclined to believe we have to remove them given the interpretation
> >>> Luciano has put forth.
> >>>
> >>> Sean
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Matei Zaharia <ma...@gmail.com>.
The question is just whether the metadata and instructions involving these Maven packages counts as sufficient to tell the user that they have different licensing terms. For example, our Ganglia package was called spark-ganglia-lgpl (so you'd notice it's a different license even from its name), and our Kinesis one was called spark-streaming-kinesis-asl, and our docs both mentioned these were under different licensing terms. But is that enough? That's the question.
Matei
> On Sep 7, 2016, at 2:05 PM, Cody Koeninger <co...@koeninger.org> wrote:
>
> To be clear, "safe" has very little to do with this.
>
> It's pretty clear that there's very little risk of the spark module
> for kinesis being considered a derivative work, much less all of
> spark.
>
> The use limitation in 3.3 that caused the amazon license to be put on
> the apache X list also doesn't have anything to do with a legal safety
> risk here. Really, what are you going to use a kinesis connector for,
> except for connecting to kinesis?
>
>
> On Wed, Sep 7, 2016 at 2:41 PM, Luciano Resende <lu...@gmail.com> wrote:
>>
>>
>> On Wed, Sep 7, 2016 at 12:20 PM, Mridul Muralidharan <mr...@gmail.com>
>> wrote:
>>>
>>>
>>> It is good to get clarification, but the way I read it, the issue is
>>> whether we publish it as official Apache artifacts (in maven, etc).
>>>
>>> Users can of course build it directly (and we can make it easy to do so) -
>>> as they are explicitly agreeing to additional licenses.
>>>
>>> Regards
>>> Mridul
>>>
>>
>> +1, by providing instructions on how the user would build, and attaching the
>> license details on the instructions, we are then safe on the legal aspects
>> of it.
>>
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Cody Koeninger <co...@koeninger.org>.
To be clear, "safe" has very little to do with this.
It's pretty clear that there's very little risk of the spark module
for kinesis being considered a derivative work, much less all of
spark.
The use limitation in 3.3 that caused the amazon license to be put on
the apache X list also doesn't have anything to do with a legal safety
risk here. Really, what are you going to use a kinesis connector for,
except for connecting to kinesis?
On Wed, Sep 7, 2016 at 2:41 PM, Luciano Resende <lu...@gmail.com> wrote:
>
>
> On Wed, Sep 7, 2016 at 12:20 PM, Mridul Muralidharan <mr...@gmail.com>
> wrote:
>>
>>
>> It is good to get clarification, but the way I read it, the issue is
>> whether we publish it as official Apache artifacts (in maven, etc).
>>
>> Users can of course build it directly (and we can make it easy to do so) -
>> as they are explicitly agreeing to additional licenses.
>>
>> Regards
>> Mridul
>>
>
> +1, by providing instructions on how the user would build, and attaching the
> license details on the instructions, we are then safe on the legal aspects
> of it.
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Luciano Resende <lu...@gmail.com>.
On Wed, Sep 7, 2016 at 12:20 PM, Mridul Muralidharan <mr...@gmail.com>
wrote:
>
> It is good to get clarification, but the way I read it, the issue is
> whether we publish it as official Apache artifacts (in maven, etc).
>
> Users can of course build it directly (and we can make it easy to do so) -
> as they are explicitly agreeing to additional licenses.
>
> Regards
> Mridul
>
>
+1, by providing instructions on how the user would build, and attaching
the license details on the instructions, we are then safe on the legal
aspects of it.
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Mridul Muralidharan <mr...@gmail.com>.
It is good to get clarification, but the way I read it, the issue is
whether we publish it as official Apache artifacts (in maven, etc).
Users can of course build it directly (and we can make it easy to do so) -
as they are explicitly agreeing to additional licenses.
Regards
Mridul
On Wednesday, September 7, 2016, Matei Zaharia <ma...@gmail.com>
wrote:
> I think you should ask legal about how to have some Maven artifacts for
> these. Both Ganglia and Kinesis are very widely used, so it's weird to ask
> users to build them from source. Maybe the Maven artifacts can be marked as
> being under a different license?
>
> In the initial discussion for LEGAL-198, we were told the following:
>
> "If the component that uses this dependency is not required for the rest
> of Spark to function then you can have a subproject to build the component.
> See http://www.apache.org/legal/resolved.html#optional. This means you
> will have to provide instructions for users to enable the optional
> component (which IMO should provide pointers to the licensing)."
>
> It's not clear whether "enable the optional component" means "every user
> must build it from source", or whether we could tell users "here's a Maven
> coordinate you can add to your project if you're okay with the licensing".
>
> Matei
>
> > On Sep 7, 2016, at 11:35 AM, Sean Owen <sowen@cloudera.com
> <javascript:;>> wrote:
> >
> > (Credit to Luciano for pointing it out)
> >
> > Yes it's clear why the assembly can't be published but I had the same
> > question about the non-assembly Kinesis (and ganglia) artifact,
> > because the published artifact has no code from Kinesis.
> >
> > See the related discussion at
> > https://issues.apache.org/jira/browse/LEGAL-198 ; the point I took
> > from there is that the Spark Kinesis artifact is optional with respect
> > to Spark, but still something published by Spark, and it requires the
> > Amazon-licensed code non-optionally.
> >
> > I'll just ask that question to confirm or deny.
> >
> > (It also has some background on why the Amazon License is considered
> > "Category X" in ASF policy due to field of use restrictions. I myself
> > take that as read rather than know the details of that decision.)
> >
> > On Wed, Sep 7, 2016 at 6:44 PM, Cody Koeninger <cody@koeninger.org
> <javascript:;>> wrote:
> >> I don't see a reason to remove the non-assembly artifact, why would
> >> you? You're not distributing copies of Amazon licensed code, and the
> >> Amazon license goes out of its way not to over-reach regarding
> >> derivative works.
> >>
> >> This seems pretty clearly to fall in the spirit of
> >>
> >> http://www.apache.org/legal/resolved.html#optional
> >>
> >> I certainly think the majority of Spark users will still want to use
> >> Spark without adding Kinesis
> >>
> >> On Wed, Sep 7, 2016 at 3:29 AM, Sean Owen <sowen@cloudera.com
> <javascript:;>> wrote:
> >>> It's worth calling attention to:
> >>>
> >>> https://issues.apache.org/jira/browse/SPARK-17418
> >>> https://issues.apache.org/jira/browse/SPARK-17422
> >>>
> >>> It looks like we need to at least not publish the kinesis *assembly*
> >>> Maven artifact because it contains Amazon Software Licensed-code
> >>> directly.
> >>>
> >>> However there's a reasonably strong reason to believe that we'd have
> >>> to remove the non-assembly Kinesis artifact too, as well as the
> >>> Ganglia one. This doesn't mean it goes away from the project, just
> >>> means it would no longer be published as a Maven artifact. (These have
> >>> never been bundled in the main Spark artifacts.)
> >>>
> >>> I wanted to give a heads up to see if anyone a) believes this
> >>> conclusion is wrong or b) wants to take it up with legal@? I'm
> >>> inclined to believe we have to remove them given the interpretation
> >>> Luciano has put forth.
> >>>
> >>> Sean
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <javascript:;>
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <javascript:;>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org <javascript:;>
>
>
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Sean Owen <so...@cloudera.com>.
Agree, I've asked the question on that thread and will follow it up.
I'd prefer not to pull these unless it's fairly clear it's going to be
against policy.
On Wed, Sep 7, 2016 at 7:57 PM, Matei Zaharia <ma...@gmail.com> wrote:
> I think you should ask legal about how to have some Maven artifacts for these. Both Ganglia and Kinesis are very widely used, so it's weird to ask users to build them from source. Maybe the Maven artifacts can be marked as being under a different license?
>
> In the initial discussion for LEGAL-198, we were told the following:
>
> "If the component that uses this dependency is not required for the rest of Spark to function then you can have a subproject to build the component. See http://www.apache.org/legal/resolved.html#optional. This means you will have to provide instructions for users to enable the optional component (which IMO should provide pointers to the licensing)."
>
> It's not clear whether "enable the optional component" means "every user must build it from source", or whether we could tell users "here's a Maven coordinate you can add to your project if you're okay with the licensing".
>
> Matei
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Matei Zaharia <ma...@gmail.com>.
I think you should ask legal about how to have some Maven artifacts for these. Both Ganglia and Kinesis are very widely used, so it's weird to ask users to build them from source. Maybe the Maven artifacts can be marked as being under a different license?
In the initial discussion for LEGAL-198, we were told the following:
"If the component that uses this dependency is not required for the rest of Spark to function then you can have a subproject to build the component. See http://www.apache.org/legal/resolved.html#optional. This means you will have to provide instructions for users to enable the optional component (which IMO should provide pointers to the licensing)."
It's not clear whether "enable the optional component" means "every user must build it from source", or whether we could tell users "here's a Maven coordinate you can add to your project if you're okay with the licensing".
Matei
> On Sep 7, 2016, at 11:35 AM, Sean Owen <so...@cloudera.com> wrote:
>
> (Credit to Luciano for pointing it out)
>
> Yes it's clear why the assembly can't be published but I had the same
> question about the non-assembly Kinesis (and ganglia) artifact,
> because the published artifact has no code from Kinesis.
>
> See the related discussion at
> https://issues.apache.org/jira/browse/LEGAL-198 ; the point I took
> from there is that the Spark Kinesis artifact is optional with respect
> to Spark, but still something published by Spark, and it requires the
> Amazon-licensed code non-optionally.
>
> I'll just ask that question to confirm or deny.
>
> (It also has some background on why the Amazon License is considered
> "Category X" in ASF policy due to field of use restrictions. I myself
> take that as read rather than know the details of that decision.)
>
> On Wed, Sep 7, 2016 at 6:44 PM, Cody Koeninger <co...@koeninger.org> wrote:
>> I don't see a reason to remove the non-assembly artifact, why would
>> you? You're not distributing copies of Amazon licensed code, and the
>> Amazon license goes out of its way not to over-reach regarding
>> derivative works.
>>
>> This seems pretty clearly to fall in the spirit of
>>
>> http://www.apache.org/legal/resolved.html#optional
>>
>> I certainly think the majority of Spark users will still want to use
>> Spark without adding Kinesis
>>
>> On Wed, Sep 7, 2016 at 3:29 AM, Sean Owen <so...@cloudera.com> wrote:
>>> It's worth calling attention to:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-17418
>>> https://issues.apache.org/jira/browse/SPARK-17422
>>>
>>> It looks like we need to at least not publish the kinesis *assembly*
>>> Maven artifact because it contains Amazon Software Licensed-code
>>> directly.
>>>
>>> However there's a reasonably strong reason to believe that we'd have
>>> to remove the non-assembly Kinesis artifact too, as well as the
>>> Ganglia one. This doesn't mean it goes away from the project, just
>>> means it would no longer be published as a Maven artifact. (These have
>>> never been bundled in the main Spark artifacts.)
>>>
>>> I wanted to give a heads up to see if anyone a) believes this
>>> conclusion is wrong or b) wants to take it up with legal@? I'm
>>> inclined to believe we have to remove them given the interpretation
>>> Luciano has put forth.
>>>
>>> Sean
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Sean Owen <so...@cloudera.com>.
(Credit to Luciano for pointing it out)
Yes it's clear why the assembly can't be published but I had the same
question about the non-assembly Kinesis (and ganglia) artifact,
because the published artifact has no code from Kinesis.
See the related discussion at
https://issues.apache.org/jira/browse/LEGAL-198 ; the point I took
from there is that the Spark Kinesis artifact is optional with respect
to Spark, but still something published by Spark, and it requires the
Amazon-licensed code non-optionally.
I'll just ask that question to confirm or deny.
(It also has some background on why the Amazon License is considered
"Category X" in ASF policy due to field of use restrictions. I myself
take that as read rather than know the details of that decision.)
On Wed, Sep 7, 2016 at 6:44 PM, Cody Koeninger <co...@koeninger.org> wrote:
> I don't see a reason to remove the non-assembly artifact, why would
> you? You're not distributing copies of Amazon licensed code, and the
> Amazon license goes out of its way not to over-reach regarding
> derivative works.
>
> This seems pretty clearly to fall in the spirit of
>
> http://www.apache.org/legal/resolved.html#optional
>
> I certainly think the majority of Spark users will still want to use
> Spark without adding Kinesis
>
> On Wed, Sep 7, 2016 at 3:29 AM, Sean Owen <so...@cloudera.com> wrote:
>> It's worth calling attention to:
>>
>> https://issues.apache.org/jira/browse/SPARK-17418
>> https://issues.apache.org/jira/browse/SPARK-17422
>>
>> It looks like we need to at least not publish the kinesis *assembly*
>> Maven artifact because it contains Amazon Software Licensed-code
>> directly.
>>
>> However there's a reasonably strong reason to believe that we'd have
>> to remove the non-assembly Kinesis artifact too, as well as the
>> Ganglia one. This doesn't mean it goes away from the project, just
>> means it would no longer be published as a Maven artifact. (These have
>> never been bundled in the main Spark artifacts.)
>>
>> I wanted to give a heads up to see if anyone a) believes this
>> conclusion is wrong or b) wants to take it up with legal@? I'm
>> inclined to believe we have to remove them given the interpretation
>> Luciano has put forth.
>>
>> Sean
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Removing published kinesis, ganglia artifacts due to license issues?
Posted by Cody Koeninger <co...@koeninger.org>.
I don't see a reason to remove the non-assembly artifact, why would
you? You're not distributing copies of Amazon licensed code, and the
Amazon license goes out of its way not to over-reach regarding
derivative works.
This seems pretty clearly to fall in the spirit of
http://www.apache.org/legal/resolved.html#optional
I certainly think the majority of Spark users will still want to use
Spark without adding Kinesis
On Wed, Sep 7, 2016 at 3:29 AM, Sean Owen <so...@cloudera.com> wrote:
> It's worth calling attention to:
>
> https://issues.apache.org/jira/browse/SPARK-17418
> https://issues.apache.org/jira/browse/SPARK-17422
>
> It looks like we need to at least not publish the kinesis *assembly*
> Maven artifact because it contains Amazon Software Licensed-code
> directly.
>
> However there's a reasonably strong reason to believe that we'd have
> to remove the non-assembly Kinesis artifact too, as well as the
> Ganglia one. This doesn't mean it goes away from the project, just
> means it would no longer be published as a Maven artifact. (These have
> never been bundled in the main Spark artifacts.)
>
> I wanted to give a heads up to see if anyone a) believes this
> conclusion is wrong or b) wants to take it up with legal@? I'm
> inclined to believe we have to remove them given the interpretation
> Luciano has put forth.
>
> Sean
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org