You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@superset.apache.org by Maxime Beauchemin <ma...@gmail.com> on 2019/01/04 17:29:24 UTC
[VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Hello,
This is not an attempt to make an official ASF release, but just to push
what we have in 0.29.0rc7 as 0.29.0 on Pypi.
The latest we have in Pypi is 0.28.1 and it's old and not a great release.
The question is whether it's ok to make a non-ASF release at this time?
I'm thinking we can start another thread about making an ASF release in
parallel.
Max
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Maxime Beauchemin <ma...@gmail.com>.
I removed the `unidecode` dep here:
https://github.com/apache/incubator-superset/pull/6673, I don't think it's
needed anymore in py3 (was it ever really needed?). Context here
https://github.com/apache/incubator-superset/pull/3975
I'm not against RAT personally, I just couldn't get it to work for me. I
looked into contributing and looked so unfamiliar and untouched (no PR on
Github, mirrored from SVN, ...) that I wrote something quick. The most
recent commit on RAT is 2012. I like the idea that it's a standard though,
then we can be RAT-compliant.
Max
On Sat, Jan 12, 2019 at 2:07 PM Bolke de Bruin <bd...@gmail.com> wrote:
> Hi,
>
> I opened a two PRs:
>
> - https://github.com/apache/incubator-superset/pull/6671 <
> https://github.com/apache/incubator-superset/pull/6671> , adds a NOTICE
> file and adds the obvious 3rd party dependencies for the *source*
> distribution
> - https://github.com/apache/incubator-superset/pull/6663 <
> https://github.com/apache/incubator-superset/pull/6663> , which adds
> Apache Rat to the build matrix and fails if *new* files are added without
> proper licenses
>
> A source release will be easier than a binary release. This is due to the
> fact that for the binary release Superset needs to include a lot of third
> party packages, which might be addressed in #5801. #5801 has issues
> unfortunately as it allows LGPL and GPL which is a no go. For the source
> release an INSTALL file is required.
>
> Also I noticed that the requirements file lists at least one GPL
> dependency 'unidecode==1.0.22’. I suggest removing it or using
> ’text-unidecode’ which is Perl Artistic. You cannot include GPL
> dependencies.
>
> I have made Apache Rat work on the code base. Max, I know you worked on
> ‘rodent’ but it is incomplete as of yet. You know I like your ‘kids’, but
> maybe this one you should not adopt ;-)? Also Rat is understood by the IPMC
> which just makes the process smoother.
>
> Cheers
> Bolke
>
>
>
> > On 7 Jan 2019, at 22:45, Maxime Beauchemin <ma...@gmail.com>
> wrote:
> >
> > Hi Bolke,
> >
> > Thanks for the thoughtful notes here. I like the idea of sticking to
> > release candidate or "devN" packages on pypi for this, personally I think
> > that's acceptable.
> >
> > About headers, I have a PR out here
> > https://github.com/apache/incubator-superset/pull/5800 <
> https://github.com/apache/incubator-superset/pull/5800>. I ended up
> writing
> > a small utility that does kind of what RAT does (called "rodent" :) as I
> > couldn't get RAT to work properly
> >
> > About LICENSE.txt, and legal/licenses aspects, we're super committed to
> > doing it, we just haven't come up with a solid plan + timeline yet. I
> think
> > realistically it can be done this quarter. I did some work here
> > https://github.com/apache/incubator-superset/pull/5801 <
> https://github.com/apache/incubator-superset/pull/5801>. From my
> > understanding some of the concerns I'm trying to address in this PR are
> not
> > a real concern for a source release where the user would fetch and build
> > the JS out of NPM. I'm not sure whether they are a concern for a
> > convenience release but I think they are, unless we find a way to *not*
> > bundle the JS "binaries" in the convenience release. The idea I had was
> to
> > commit a whitelist of packages that have non-standard license on NPM
> > (things like "apache2*" or "MIT*", or "MIT with LLVN clause") in the repo
> > and to have some automated checks to raise issues when new,
> non-whitelisted
> > libs with non-standard licenses would show up on the dependency tree.
> >
> > Max
> >
> > On Mon, Jan 7, 2019 at 12:07 AM Bolke de Bruin <bdbruin@gmail.com
> <ma...@gmail.com>> wrote:
> >
> >> Hi All,
> >>
> >> (Part of the thread seems missing, but I assume everyone has the same
> >> history).
> >>
> >> Just hopping over from Apache Airflow. We have been going through the
> same
> >> process and some of you even have been involved :-). We have also
> struggled
> >> along the way, but we found our path and have just graduated from the
> >> incubator last month. We are also a Python based project.
> >>
> >> I personally reach out as I made the company I work for dependent on
> >> Superset, but we cannot continue to use it if it doesn’t eventually
> does a
> >> legal compliant release. During incubation the ASF puts you through the
> >> mill in order to get to such a release, but others like CNCF will
> require
> >> the same.
> >>
> >> So here to share some experiences, some suggestions, and offer some
> help.
> >>
> >> On the ‘VOTE’
> >>
> >> Releases in Apache’s world are a very loaded concept. It is not so much
> >> about bug fixes and new features, but much more if those new features
> and
> >> bug fixes fit in the legal framework. Primarily are copyrights correct
> and
> >> licenses compliant? During voting it is assumed the the PMC takes a
> serious
> >> look at this. To help, during incubation, the IPMC will also take a
> serious
> >> look at this. Hence the double voting.
> >>
> >> If you come from outside the incubator (as Superset did and also Airflow
> >> did) then this process feels slow and bureaucratic. You releases when
> you
> >> wanted and no autonomy is taken away. You want to cater to your users
> and
> >> fix bugs quickly. Going back and forth to the IPMC is a burden. To keep
> >> the momentum while (72h + 72h) * X times voting is tough. Particularly,
> if
> >> you are doing this on a voluntary basis.
> >>
> >> Now your users, that have grown beyond before you joined the incubator,
> >> trust the Apache brand and all that comes with it. Actually, because of
> the
> >> Apache brand some corporates (like mine) can now (finally) use Superset
> as
> >> we trust the process Apache puts in place.
> >>
> >> This vote doesn’t do that and you are not doing your users a favor with
> >> it. Moreover, you put them at risk of becoming legally liable. You can
> >> argue that having rc7 on PyPi for developer convenience is fine. ‘Pip’
> does
> >> not install rc’s (or betas etc) without asking it to do so explicitly.
> It
> >> kind of works as Maven snapshots in the java world. So installation only
> >> happens when you know what you are doing and thus understand the extra
> risk
> >> that you are taking with installing it (functionally and legally).
> Justin
> >> might not entirely agree however ;-).
> >>
> >> So it is fine to point *developers* to a place where they can pick up
> what
> >> they need to further develop Superset. If that involves company wide
> >> testing so be it. It is assumed they know what they are doing. However,
> you
> >> cannot point you users to it. They have different expectations.
> >>
> >> Suggestion:
> >>
> >> 1) do not rename, everyone who needs development features and bug fixes
> >> can install the development version so they can use that for testing.
> >> 2) do not misuse “release”
> >>
> >> LICENSES
> >>
> >> I noticed that a lot of the files do not have ASF headers. I’m also
> pretty
> >> sure that Superset has quite a lot third party dependencies that come
> with
> >> their own licenses[1]. There is nothing in LICENSE.txt that points to
> those.
> >>
> >> Yes, it is not fun to track down all those licenses (and versions).
> >> However, it is mostly a one-off job. It only requires updating when you
> >> change or update the dependency. To a certain extend it can be automated
> >> even!
> >>
> >> Apache has some software that helps with license compliancy. Apache RAT
> >> works well and in Airflow we created a script that does this as part of
> our
> >> CI[1]. What we did (see git history) is to keep a count of the amount of
> >> “unknown” licenses and that count was not allowed to go up. I think we
> even
> >> did a force of -1 at some point to require PRs to add license headers
> so we
> >> didnt have to face that burden entirely on our own. How’s that for crowd
> >> sourcing :-).
> >>
> >> Suggestion:
> >>
> >> 1) Add ASF headers to all files that should have it. Do this as a
> one-off
> >> semi-automation
> >> 2) Add Apache RAT as part of your build process and fail PRs if they
> don’t
> >> pass.
> >>
> >>
> >> MY HELP:
> >>
> >> Let me see if I can add Apache Rat.
> >>
> >>
> >> Hope this helps!
> >>
> >> Bolke
> >>
> >>
> >> [1] https://github.com/syntagmatic/parallel-coordinates <
> >> https://github.com/syntagmatic/parallel-coordinates <
> https://github.com/syntagmatic/parallel-coordinates>>
> >> [2]
> >>
> https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh
> <
> https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh
> >
> >> <
> >>
> https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh
> <
> https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh
> >
> >>>
> >>
> >>
> >>> Op 6 jan. 2019, om 22:02 heeft Justin Mclean <jm...@apache.org> het
> >> volgende geschreven:
> >>>
> >>> Hi,
> >>>
> >>> Release policy [1] states:
> >>> "Projects MUST direct outsiders towards official releases rather than
> >> raw source repositories, nightly builds, snapshots, release candidates,
> or
> >> any other similar packages."
> >>>
> >>> In this context an official release is an ASF source release vetted
> >> (usually by voting) by the PPMC.
> >>>
> >>> See also (for instance) [2] - "It is appropriate to distribute official
> >> releases through downstream channels, but inappropriate to distribute
> >> unreleased materials through them."
> >>>
> >>> IMO you would need permission from VP legal to be able make further
> >> unapproved release. I'd ask on the legal-discuss@ mailing list.
> >>>
> >>> Your mentors may have different opinion or some other advice to give.
> >>>
> >>> Thanks,
> >>> Justin
> >>>
> >>> 1. http://www.apache.org/legal/release-policy.html#publication
> >>> 2. https://issues.apache.org/jira/browse/LEGAL-270
>
>
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi,
I opened a two PRs:
- https://github.com/apache/incubator-superset/pull/6671 <https://github.com/apache/incubator-superset/pull/6671> , adds a NOTICE file and adds the obvious 3rd party dependencies for the *source* distribution
- https://github.com/apache/incubator-superset/pull/6663 <https://github.com/apache/incubator-superset/pull/6663> , which adds Apache Rat to the build matrix and fails if *new* files are added without proper licenses
A source release will be easier than a binary release. This is due to the fact that for the binary release Superset needs to include a lot of third party packages, which might be addressed in #5801. #5801 has issues unfortunately as it allows LGPL and GPL which is a no go. For the source release an INSTALL file is required.
Also I noticed that the requirements file lists at least one GPL dependency 'unidecode==1.0.22’. I suggest removing it or using ’text-unidecode’ which is Perl Artistic. You cannot include GPL dependencies.
I have made Apache Rat work on the code base. Max, I know you worked on ‘rodent’ but it is incomplete as of yet. You know I like your ‘kids’, but maybe this one you should not adopt ;-)? Also Rat is understood by the IPMC which just makes the process smoother.
Cheers
Bolke
> On 7 Jan 2019, at 22:45, Maxime Beauchemin <ma...@gmail.com> wrote:
>
> Hi Bolke,
>
> Thanks for the thoughtful notes here. I like the idea of sticking to
> release candidate or "devN" packages on pypi for this, personally I think
> that's acceptable.
>
> About headers, I have a PR out here
> https://github.com/apache/incubator-superset/pull/5800 <https://github.com/apache/incubator-superset/pull/5800>. I ended up writing
> a small utility that does kind of what RAT does (called "rodent" :) as I
> couldn't get RAT to work properly
>
> About LICENSE.txt, and legal/licenses aspects, we're super committed to
> doing it, we just haven't come up with a solid plan + timeline yet. I think
> realistically it can be done this quarter. I did some work here
> https://github.com/apache/incubator-superset/pull/5801 <https://github.com/apache/incubator-superset/pull/5801>. From my
> understanding some of the concerns I'm trying to address in this PR are not
> a real concern for a source release where the user would fetch and build
> the JS out of NPM. I'm not sure whether they are a concern for a
> convenience release but I think they are, unless we find a way to *not*
> bundle the JS "binaries" in the convenience release. The idea I had was to
> commit a whitelist of packages that have non-standard license on NPM
> (things like "apache2*" or "MIT*", or "MIT with LLVN clause") in the repo
> and to have some automated checks to raise issues when new, non-whitelisted
> libs with non-standard licenses would show up on the dependency tree.
>
> Max
>
> On Mon, Jan 7, 2019 at 12:07 AM Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>
>> Hi All,
>>
>> (Part of the thread seems missing, but I assume everyone has the same
>> history).
>>
>> Just hopping over from Apache Airflow. We have been going through the same
>> process and some of you even have been involved :-). We have also struggled
>> along the way, but we found our path and have just graduated from the
>> incubator last month. We are also a Python based project.
>>
>> I personally reach out as I made the company I work for dependent on
>> Superset, but we cannot continue to use it if it doesn’t eventually does a
>> legal compliant release. During incubation the ASF puts you through the
>> mill in order to get to such a release, but others like CNCF will require
>> the same.
>>
>> So here to share some experiences, some suggestions, and offer some help.
>>
>> On the ‘VOTE’
>>
>> Releases in Apache’s world are a very loaded concept. It is not so much
>> about bug fixes and new features, but much more if those new features and
>> bug fixes fit in the legal framework. Primarily are copyrights correct and
>> licenses compliant? During voting it is assumed the the PMC takes a serious
>> look at this. To help, during incubation, the IPMC will also take a serious
>> look at this. Hence the double voting.
>>
>> If you come from outside the incubator (as Superset did and also Airflow
>> did) then this process feels slow and bureaucratic. You releases when you
>> wanted and no autonomy is taken away. You want to cater to your users and
>> fix bugs quickly. Going back and forth to the IPMC is a burden. To keep
>> the momentum while (72h + 72h) * X times voting is tough. Particularly, if
>> you are doing this on a voluntary basis.
>>
>> Now your users, that have grown beyond before you joined the incubator,
>> trust the Apache brand and all that comes with it. Actually, because of the
>> Apache brand some corporates (like mine) can now (finally) use Superset as
>> we trust the process Apache puts in place.
>>
>> This vote doesn’t do that and you are not doing your users a favor with
>> it. Moreover, you put them at risk of becoming legally liable. You can
>> argue that having rc7 on PyPi for developer convenience is fine. ‘Pip’ does
>> not install rc’s (or betas etc) without asking it to do so explicitly. It
>> kind of works as Maven snapshots in the java world. So installation only
>> happens when you know what you are doing and thus understand the extra risk
>> that you are taking with installing it (functionally and legally). Justin
>> might not entirely agree however ;-).
>>
>> So it is fine to point *developers* to a place where they can pick up what
>> they need to further develop Superset. If that involves company wide
>> testing so be it. It is assumed they know what they are doing. However, you
>> cannot point you users to it. They have different expectations.
>>
>> Suggestion:
>>
>> 1) do not rename, everyone who needs development features and bug fixes
>> can install the development version so they can use that for testing.
>> 2) do not misuse “release”
>>
>> LICENSES
>>
>> I noticed that a lot of the files do not have ASF headers. I’m also pretty
>> sure that Superset has quite a lot third party dependencies that come with
>> their own licenses[1]. There is nothing in LICENSE.txt that points to those.
>>
>> Yes, it is not fun to track down all those licenses (and versions).
>> However, it is mostly a one-off job. It only requires updating when you
>> change or update the dependency. To a certain extend it can be automated
>> even!
>>
>> Apache has some software that helps with license compliancy. Apache RAT
>> works well and in Airflow we created a script that does this as part of our
>> CI[1]. What we did (see git history) is to keep a count of the amount of
>> “unknown” licenses and that count was not allowed to go up. I think we even
>> did a force of -1 at some point to require PRs to add license headers so we
>> didnt have to face that burden entirely on our own. How’s that for crowd
>> sourcing :-).
>>
>> Suggestion:
>>
>> 1) Add ASF headers to all files that should have it. Do this as a one-off
>> semi-automation
>> 2) Add Apache RAT as part of your build process and fail PRs if they don’t
>> pass.
>>
>>
>> MY HELP:
>>
>> Let me see if I can add Apache Rat.
>>
>>
>> Hope this helps!
>>
>> Bolke
>>
>>
>> [1] https://github.com/syntagmatic/parallel-coordinates <
>> https://github.com/syntagmatic/parallel-coordinates <https://github.com/syntagmatic/parallel-coordinates>>
>> [2]
>> https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh <https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh>
>> <
>> https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh <https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh>
>>>
>>
>>
>>> Op 6 jan. 2019, om 22:02 heeft Justin Mclean <jm...@apache.org> het
>> volgende geschreven:
>>>
>>> Hi,
>>>
>>> Release policy [1] states:
>>> "Projects MUST direct outsiders towards official releases rather than
>> raw source repositories, nightly builds, snapshots, release candidates, or
>> any other similar packages."
>>>
>>> In this context an official release is an ASF source release vetted
>> (usually by voting) by the PPMC.
>>>
>>> See also (for instance) [2] - "It is appropriate to distribute official
>> releases through downstream channels, but inappropriate to distribute
>> unreleased materials through them."
>>>
>>> IMO you would need permission from VP legal to be able make further
>> unapproved release. I'd ask on the legal-discuss@ mailing list.
>>>
>>> Your mentors may have different opinion or some other advice to give.
>>>
>>> Thanks,
>>> Justin
>>>
>>> 1. http://www.apache.org/legal/release-policy.html#publication
>>> 2. https://issues.apache.org/jira/browse/LEGAL-270
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Justin Mclean <jm...@apache.org>.
Hi,
> https://www.linuxfoundation.org/press-release/2018/12/the-linux-foundation-to-launch-new-tooling-project-to-improve-open-source-compliance/
It's based on SPDX [1] and I'm guessing t's unlikely that all that 3rd party JS files have SPDX IDs. In other word you need to create them which is probably a lot more work.
Thanks,
Justin
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Maxime Beauchemin <ma...@gmail.com>.
On the topic of compliance, LF just announced new tooling to help automate
things around compliance. That looks pretty compelling to me. I'm not sure
how much of this tooling is open and we could just leverage while at the
ASF.
https://www.linuxfoundation.org/press-release/2018/12/the-linux-foundation-to-launch-new-tooling-project-to-improve-open-source-compliance/
On Tue, Jan 8, 2019 at 4:12 PM Maxime Beauchemin <ma...@gmail.com>
wrote:
> Thanks for the input Justin, we should refrain from pushing anything to
> Pypi until we figure things out then. The RC process should only rely on
> references to Github tags.
>
> We'll come up with a plan for the first ASF code release. My impression is
> that it'll be easier once the plugin / embeddable components work wraps up.
>
> I have a question about "companion" convenience release. So say we make a
> source release and it gets voted on approved, signed and properly released.
> Now if we want to make a companion release to Pypi, how should we go about
> that? Also if we want to ship the javascript bundles along with the Pypi
> tarball (not bundled in code releases but convenient to have in Pypi
> releases), do we need to have another LICENSE.txt listing out the 577
> packages we use in our production JS bundles, sorted out by license? If so,
> what are alternatives? Would it be ok to add a "superset build" command or
> a "superset download-js-bundle" to run as part of the installation process?
>
> Max
>
> On Tue, Jan 8, 2019 at 12:50 PM Justin Mclean <jm...@apache.org> wrote:
>
>> Hi,
>>
>> > Thanks for the thoughtful notes here. I like the idea of sticking to
>> > release candidate or "devN" packages on pypi for this, personally I
>> think
>> > that's acceptable.
>>
>> I suggest you reread the links I posted above, making release candidates
>> available to the general public is not line line with ASF policy, again if
>> you want to do that IMO you would need permission from VP legal.
>>
>> > About LICENSE.txt, and legal/licenses aspects, we're super committed to
>> > doing it, we just haven't come up with a solid plan + timeline yet. I
>> think
>> > realistically it can be done this quarter.
>>
>> This work would would take a few hours and perhaps a couple of iterations
>> of release candidates to get right worse case so I'm not sure why you need
>> 4 months. II suggest you start by making a source only release, which is
>> what ASF projects actually release. A quick search of your git repo shows a
>> single BSD licensed 3rd party file and creating a LICENSE and NOTICE for
>> that is trivial. Are there any files outside the ASF repo that need to go
>> into a release?
>>
>> Thanks,
>> Justin
>>
>
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Justin Mclean <jm...@apache.org>.
Hi,
> I have a question about "companion" convenience release. So say we make a
> source release and it gets voted on approved, signed and properly released.
> Now if we want to make a companion release to Pypi, how should we go about
> that?
There are a number of ways of dealing with it, what usually happens is that both the source and convenance binary are made at the same time and voted on. Both need to follow policy but it more important that the source does, being an incubating project there is some leeway because of your "DISCAIMER". For a first release it might be acceptable to have a good source release and the connivence binary with a note in it license saying that "a full list of all licenses are still being compiled use at your own risk". However this would need to be fixed in later releases and before graduation.
>Also if we want to ship the javascript bundles along with the Pypi
> tarball (not bundled in code releases but convenient to have in Pypi
> releases), do we need to have another LICENSE.txt listing out the 577
> packages we use in our production JS bundles, sorted out by license?
Yes the convenance binary would need to list anything else that is bundled in the release. Usually what projects do is have a pointer to the full license text in the LICENSE file.
> If so, what are alternatives? Would it be ok to add a "superset build" command or
> a "superset download-js-bundle" to run as part of the installation process?
That might also be acceptable, but I have some doubts. you would need to ask on legal-discuss. Mostly because the user now does not know the terms and conditions they can use the software under. Presumably all are ALv2 compatible. Think of the typical Java project that has a pom.xml listing a large number of dependancies, those do not have to be listed in the source release but those jars are usually included in the binary so would need to be listed. Not including the jars seems to be side stepping the issue.
Thanks,
Justin
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Maxime Beauchemin <ma...@gmail.com>.
Thanks for the input Justin, we should refrain from pushing anything to
Pypi until we figure things out then. The RC process should only rely on
references to Github tags.
We'll come up with a plan for the first ASF code release. My impression is
that it'll be easier once the plugin / embeddable components work wraps up.
I have a question about "companion" convenience release. So say we make a
source release and it gets voted on approved, signed and properly released.
Now if we want to make a companion release to Pypi, how should we go about
that? Also if we want to ship the javascript bundles along with the Pypi
tarball (not bundled in code releases but convenient to have in Pypi
releases), do we need to have another LICENSE.txt listing out the 577
packages we use in our production JS bundles, sorted out by license? If so,
what are alternatives? Would it be ok to add a "superset build" command or
a "superset download-js-bundle" to run as part of the installation process?
Max
On Tue, Jan 8, 2019 at 12:50 PM Justin Mclean <jm...@apache.org> wrote:
> Hi,
>
> > Thanks for the thoughtful notes here. I like the idea of sticking to
> > release candidate or "devN" packages on pypi for this, personally I think
> > that's acceptable.
>
> I suggest you reread the links I posted above, making release candidates
> available to the general public is not line line with ASF policy, again if
> you want to do that IMO you would need permission from VP legal.
>
> > About LICENSE.txt, and legal/licenses aspects, we're super committed to
> > doing it, we just haven't come up with a solid plan + timeline yet. I
> think
> > realistically it can be done this quarter.
>
> This work would would take a few hours and perhaps a couple of iterations
> of release candidates to get right worse case so I'm not sure why you need
> 4 months. II suggest you start by making a source only release, which is
> what ASF projects actually release. A quick search of your git repo shows a
> single BSD licensed 3rd party file and creating a LICENSE and NOTICE for
> that is trivial. Are there any files outside the ASF repo that need to go
> into a release?
>
> Thanks,
> Justin
>
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Justin Mclean <jm...@apache.org>.
Hi,
> Thanks for the thoughtful notes here. I like the idea of sticking to
> release candidate or "devN" packages on pypi for this, personally I think
> that's acceptable.
I suggest you reread the links I posted above, making release candidates available to the general public is not line line with ASF policy, again if you want to do that IMO you would need permission from VP legal.
> About LICENSE.txt, and legal/licenses aspects, we're super committed to
> doing it, we just haven't come up with a solid plan + timeline yet. I think
> realistically it can be done this quarter.
This work would would take a few hours and perhaps a couple of iterations of release candidates to get right worse case so I'm not sure why you need 4 months. II suggest you start by making a source only release, which is what ASF projects actually release. A quick search of your git repo shows a single BSD licensed 3rd party file and creating a LICENSE and NOTICE for that is trivial. Are there any files outside the ASF repo that need to go into a release?
Thanks,
Justin
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Maxime Beauchemin <ma...@gmail.com>.
Hi Bolke,
Thanks for the thoughtful notes here. I like the idea of sticking to
release candidate or "devN" packages on pypi for this, personally I think
that's acceptable.
About headers, I have a PR out here
https://github.com/apache/incubator-superset/pull/5800. I ended up writing
a small utility that does kind of what RAT does (called "rodent" :) as I
couldn't get RAT to work properly
About LICENSE.txt, and legal/licenses aspects, we're super committed to
doing it, we just haven't come up with a solid plan + timeline yet. I think
realistically it can be done this quarter. I did some work here
https://github.com/apache/incubator-superset/pull/5801. From my
understanding some of the concerns I'm trying to address in this PR are not
a real concern for a source release where the user would fetch and build
the JS out of NPM. I'm not sure whether they are a concern for a
convenience release but I think they are, unless we find a way to *not*
bundle the JS "binaries" in the convenience release. The idea I had was to
commit a whitelist of packages that have non-standard license on NPM
(things like "apache2*" or "MIT*", or "MIT with LLVN clause") in the repo
and to have some automated checks to raise issues when new, non-whitelisted
libs with non-standard licenses would show up on the dependency tree.
Max
On Mon, Jan 7, 2019 at 12:07 AM Bolke de Bruin <bd...@gmail.com> wrote:
> Hi All,
>
> (Part of the thread seems missing, but I assume everyone has the same
> history).
>
> Just hopping over from Apache Airflow. We have been going through the same
> process and some of you even have been involved :-). We have also struggled
> along the way, but we found our path and have just graduated from the
> incubator last month. We are also a Python based project.
>
> I personally reach out as I made the company I work for dependent on
> Superset, but we cannot continue to use it if it doesn’t eventually does a
> legal compliant release. During incubation the ASF puts you through the
> mill in order to get to such a release, but others like CNCF will require
> the same.
>
> So here to share some experiences, some suggestions, and offer some help.
>
> On the ‘VOTE’
>
> Releases in Apache’s world are a very loaded concept. It is not so much
> about bug fixes and new features, but much more if those new features and
> bug fixes fit in the legal framework. Primarily are copyrights correct and
> licenses compliant? During voting it is assumed the the PMC takes a serious
> look at this. To help, during incubation, the IPMC will also take a serious
> look at this. Hence the double voting.
>
> If you come from outside the incubator (as Superset did and also Airflow
> did) then this process feels slow and bureaucratic. You releases when you
> wanted and no autonomy is taken away. You want to cater to your users and
> fix bugs quickly. Going back and forth to the IPMC is a burden. To keep
> the momentum while (72h + 72h) * X times voting is tough. Particularly, if
> you are doing this on a voluntary basis.
>
> Now your users, that have grown beyond before you joined the incubator,
> trust the Apache brand and all that comes with it. Actually, because of the
> Apache brand some corporates (like mine) can now (finally) use Superset as
> we trust the process Apache puts in place.
>
> This vote doesn’t do that and you are not doing your users a favor with
> it. Moreover, you put them at risk of becoming legally liable. You can
> argue that having rc7 on PyPi for developer convenience is fine. ‘Pip’ does
> not install rc’s (or betas etc) without asking it to do so explicitly. It
> kind of works as Maven snapshots in the java world. So installation only
> happens when you know what you are doing and thus understand the extra risk
> that you are taking with installing it (functionally and legally). Justin
> might not entirely agree however ;-).
>
> So it is fine to point *developers* to a place where they can pick up what
> they need to further develop Superset. If that involves company wide
> testing so be it. It is assumed they know what they are doing. However, you
> cannot point you users to it. They have different expectations.
>
> Suggestion:
>
> 1) do not rename, everyone who needs development features and bug fixes
> can install the development version so they can use that for testing.
> 2) do not misuse “release”
>
> LICENSES
>
> I noticed that a lot of the files do not have ASF headers. I’m also pretty
> sure that Superset has quite a lot third party dependencies that come with
> their own licenses[1]. There is nothing in LICENSE.txt that points to those.
>
> Yes, it is not fun to track down all those licenses (and versions).
> However, it is mostly a one-off job. It only requires updating when you
> change or update the dependency. To a certain extend it can be automated
> even!
>
> Apache has some software that helps with license compliancy. Apache RAT
> works well and in Airflow we created a script that does this as part of our
> CI[1]. What we did (see git history) is to keep a count of the amount of
> “unknown” licenses and that count was not allowed to go up. I think we even
> did a force of -1 at some point to require PRs to add license headers so we
> didnt have to face that burden entirely on our own. How’s that for crowd
> sourcing :-).
>
> Suggestion:
>
> 1) Add ASF headers to all files that should have it. Do this as a one-off
> semi-automation
> 2) Add Apache RAT as part of your build process and fail PRs if they don’t
> pass.
>
>
> MY HELP:
>
> Let me see if I can add Apache Rat.
>
>
> Hope this helps!
>
> Bolke
>
>
> [1] https://github.com/syntagmatic/parallel-coordinates <
> https://github.com/syntagmatic/parallel-coordinates>
> [2]
> https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh
> <
> https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh
> >
>
>
> > Op 6 jan. 2019, om 22:02 heeft Justin Mclean <jm...@apache.org> het
> volgende geschreven:
> >
> > Hi,
> >
> > Release policy [1] states:
> > "Projects MUST direct outsiders towards official releases rather than
> raw source repositories, nightly builds, snapshots, release candidates, or
> any other similar packages."
> >
> > In this context an official release is an ASF source release vetted
> (usually by voting) by the PPMC.
> >
> > See also (for instance) [2] - "It is appropriate to distribute official
> releases through downstream channels, but inappropriate to distribute
> unreleased materials through them."
> >
> > IMO you would need permission from VP legal to be able make further
> unapproved release. I'd ask on the legal-discuss@ mailing list.
> >
> > Your mentors may have different opinion or some other advice to give.
> >
> > Thanks,
> > Justin
> >
> > 1. http://www.apache.org/legal/release-policy.html#publication
> > 2. https://issues.apache.org/jira/browse/LEGAL-270
>
>
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Bolke de Bruin <bd...@gmail.com>.
Hi All,
(Part of the thread seems missing, but I assume everyone has the same history).
Just hopping over from Apache Airflow. We have been going through the same process and some of you even have been involved :-). We have also struggled along the way, but we found our path and have just graduated from the incubator last month. We are also a Python based project.
I personally reach out as I made the company I work for dependent on Superset, but we cannot continue to use it if it doesn’t eventually does a legal compliant release. During incubation the ASF puts you through the mill in order to get to such a release, but others like CNCF will require the same.
So here to share some experiences, some suggestions, and offer some help.
On the ‘VOTE’
Releases in Apache’s world are a very loaded concept. It is not so much about bug fixes and new features, but much more if those new features and bug fixes fit in the legal framework. Primarily are copyrights correct and licenses compliant? During voting it is assumed the the PMC takes a serious look at this. To help, during incubation, the IPMC will also take a serious look at this. Hence the double voting.
If you come from outside the incubator (as Superset did and also Airflow did) then this process feels slow and bureaucratic. You releases when you wanted and no autonomy is taken away. You want to cater to your users and fix bugs quickly. Going back and forth to the IPMC is a burden. To keep the momentum while (72h + 72h) * X times voting is tough. Particularly, if you are doing this on a voluntary basis.
Now your users, that have grown beyond before you joined the incubator, trust the Apache brand and all that comes with it. Actually, because of the Apache brand some corporates (like mine) can now (finally) use Superset as we trust the process Apache puts in place.
This vote doesn’t do that and you are not doing your users a favor with it. Moreover, you put them at risk of becoming legally liable. You can argue that having rc7 on PyPi for developer convenience is fine. ‘Pip’ does not install rc’s (or betas etc) without asking it to do so explicitly. It kind of works as Maven snapshots in the java world. So installation only happens when you know what you are doing and thus understand the extra risk that you are taking with installing it (functionally and legally). Justin might not entirely agree however ;-).
So it is fine to point *developers* to a place where they can pick up what they need to further develop Superset. If that involves company wide testing so be it. It is assumed they know what they are doing. However, you cannot point you users to it. They have different expectations.
Suggestion:
1) do not rename, everyone who needs development features and bug fixes can install the development version so they can use that for testing.
2) do not misuse “release”
LICENSES
I noticed that a lot of the files do not have ASF headers. I’m also pretty sure that Superset has quite a lot third party dependencies that come with their own licenses[1]. There is nothing in LICENSE.txt that points to those.
Yes, it is not fun to track down all those licenses (and versions). However, it is mostly a one-off job. It only requires updating when you change or update the dependency. To a certain extend it can be automated even!
Apache has some software that helps with license compliancy. Apache RAT works well and in Airflow we created a script that does this as part of our CI[1]. What we did (see git history) is to keep a count of the amount of “unknown” licenses and that count was not allowed to go up. I think we even did a force of -1 at some point to require PRs to add license headers so we didnt have to face that burden entirely on our own. How’s that for crowd sourcing :-).
Suggestion:
1) Add ASF headers to all files that should have it. Do this as a one-off semi-automation
2) Add Apache RAT as part of your build process and fail PRs if they don’t pass.
MY HELP:
Let me see if I can add Apache Rat.
Hope this helps!
Bolke
[1] https://github.com/syntagmatic/parallel-coordinates <https://github.com/syntagmatic/parallel-coordinates>
[2] https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh <https://github.com/apache/airflow/blob/master/scripts/ci/6-check-license.sh>
> Op 6 jan. 2019, om 22:02 heeft Justin Mclean <jm...@apache.org> het volgende geschreven:
>
> Hi,
>
> Release policy [1] states:
> "Projects MUST direct outsiders towards official releases rather than raw source repositories, nightly builds, snapshots, release candidates, or any other similar packages."
>
> In this context an official release is an ASF source release vetted (usually by voting) by the PPMC.
>
> See also (for instance) [2] - "It is appropriate to distribute official releases through downstream channels, but inappropriate to distribute unreleased materials through them."
>
> IMO you would need permission from VP legal to be able make further unapproved release. I'd ask on the legal-discuss@ mailing list.
>
> Your mentors may have different opinion or some other advice to give.
>
> Thanks,
> Justin
>
> 1. http://www.apache.org/legal/release-policy.html#publication
> 2. https://issues.apache.org/jira/browse/LEGAL-270
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Justin Mclean <jm...@apache.org>.
Hi,
Release policy [1] states:
"Projects MUST direct outsiders towards official releases rather than raw source repositories, nightly builds, snapshots, release candidates, or any other similar packages."
In this context an official release is an ASF source release vetted (usually by voting) by the PPMC.
See also (for instance) [2] - "It is appropriate to distribute official releases through downstream channels, but inappropriate to distribute unreleased materials through them."
IMO you would need permission from VP legal to be able make further unapproved release. I'd ask on the legal-discuss@ mailing list.
Your mentors may have different opinion or some other advice to give.
Thanks,
Justin
1. http://www.apache.org/legal/release-policy.html#publication
2. https://issues.apache.org/jira/browse/LEGAL-270
Re: [VOTE] release Superset 0.29.0rc7 as 0.29.0 to Pypi
Posted by Jeff Feng <je...@airbnb.com.INVALID>.
Given that 0.28.1 isn't a very good release and 0.29.0rc7 is a more stable
release, I would be supportive of that as it will take us some time to
organize and execute an official ASF release.
Mentors - what are your thoughts? If you are also supportive, what is the
process for getting community alignment on this? Would we need to do a
formal vote?
Jeff
On Fri, Jan 4, 2019 at 9:29 AM Maxime Beauchemin <ma...@gmail.com>
wrote:
> Hello,
>
> This is not an attempt to make an official ASF release, but just to push
> what we have in 0.29.0rc7 as 0.29.0 on Pypi.
>
> The latest we have in Pypi is 0.28.1 and it's old and not a great release.
>
> The question is whether it's ok to make a non-ASF release at this time?
>
> I'm thinking we can start another thread about making an ASF release in
> parallel.
>
> Max
>
--
*Jeff Feng*
Product Lead
m: (949)-610-5108
twitter: @jtfeng