You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@superset.apache.org by Maxime Beauchemin <ma...@gmail.com> on 2019/06/09 03:12:25 UTC
Re: [RESULT][VOTE] Release Superset 0.33.0 based on Superset 0.33.0rc1
Thank you for volunteering Ville & Gianluca! I think that would sort out
the Python side deps. Follow the process here:
https://github.com/apache/incubator-superset/pull/7643
Though as mentioned many times before, the JS side is more intricate. The
dep tree is huge and dynamic (the tree changes as we bump or alter our
direct dependency).
I updated the PR that tries to summarizes the problematic JS deps and tries
to automate the process here:
https://github.com/apache/incubator-superset/pull/5801
Here's a summary of problematic JS npm libs, organized by license
{
"Custom: https://github.com/tmcw/jsonlint": [
""
],
"MIT*": [
"optimist",
"exif-parser",
"mapbox-gl",
"expect.js",
"split",
"process",
"trim",
"typed-function"
],
"Custom: https://github.com/IonicaBizau/node-blah": [
"typpy",
"flat-colors"
],
"Custom: http://badges.github.io/stability-badges/dist/stable.svg": [
"gl-vec2",
"gl-mat3",
"gl-vec3"
],
"AFLv2.1": [],
"Apache-2.0 WITH LLVM-exception": [
"mousetrap"
],
"CC0-1.0": [
"string-hash"
],
"Unlicense": [
"tweetnacl"
]
}
I tried digging into these individually and I clearly don't know enough
about licenses and don't have the authority to say one way or the other. At
this point I don't really want to work on ASF releases unless I know
whether it's necessary or not to sort these out, or how to go about them. *Keep
in mind that those libs are not part of the source release*, they get
fetched during the build process. I was told before that this was ok, but
now I'm hearing the opposite.
I'm blocked and to be honest somewhat frustrated. I do not see a clear path
forward.
Now the last Pypi release 0.28.1 (from October 2018) is completely broken
and there's no way for us to release to our community, not even just
pushing Github tags... This is hurting us. Are there ways to mitigate this
until we sort licenses out?
Max
On Tue, May 28, 2019 at 12:41 PM Ville Brofeldt <vi...@gmail.com>
wrote:
> Hi all,
>
> I volunteer to start work on making `requests` an optional dependency. It
> would be great if this could wait until the next release as Alan suggests
> to make sure there is enough time to ensure the transition goes smoothly.
>
> Ville
>
>
> On Tue, May 28, 2019, 22:24 Alan Gates <al...@gmail.com> wrote:
>
> > Again, I don't think this has to be fixed for your first incubator
> > release. It will need to be fixed before you do a second release. And
> if
> > you were a TLP you couldn't release with this. But for now, as a
> learning
> > exercise and to get a release out ASAP, I'm ok without fixing it first.
> > Especially since there are similar issues to deal with on the JavaScript
> > side.
> >
> > Alan.
> >
> > On Tue, May 28, 2019 at 11:54 AM Maxime Beauchemin <
> > maximebeauchemin@gmail.com> wrote:
> >
> > > @community, it'd be great if someone could volunteer to do this
> (making
> > > the python library "requests" an optional dependency). It appears it's
> > used
> > > in its simplest form in a few places (requests.get) which should be
> easy
> > to
> > > replace with urrlib2. The Druid connector uses a more advanced feature
> > > (basic auth), but that could be made just an optional dependency, as
> > we're
> > > deprecating the Druid connector, to be replaced by the SQLAlchemy Druid
> > > connector/dialect. There it's just a matter of moving `pydruid` and
> > > `requests` to the "extra_requires" section of our setup.py, and doing
> > late
> > > imports.
> > >
> > > If not it may take me a moment to get to do this on top of everything
> > else
> > > needed for the ASF official release.
> > >
> > > Max
> > >
> > > On Sat, May 25, 2019 at 12:33 AM Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >
> > > > https://issues.apache.org/jira/browse/LEGAL-192
> > > >
> > > > Details why. It comes down to that you can use LGPL, but only if it’s
> > an
> > > > optional dependency. This is policy of the ASF in order not to limit
> > > > downstream usage of its own products. Ie. A Non optional dependency
> > would
> > > > require downstream usage of Superset to abide by the LGPL (“reverse
> > > > engineering”) without a way out.
> > > >
> > > > As our use of the requests modules is fairly limited I suggest to
> > > > replicate the functionality that is required. Another option is to
> fork
> > > > requests and kick out the chardet dependency which is only used in
> one
> > > > place and probably not relevant to Superset.
> > > >
> > > > Cheers
> > > > Bolke
> > > >
> > > > Verstuurd vanaf mijn iPad
> > > >
> > > > > Op 23 mei 2019 om 01:37 heeft Maxime Beauchemin <
> > > > maximebeauchemin@gmail.com> het volgende geschreven:
> > > > >
> > > > > Related, about requests/chardet
> > > > > https://github.com/kennethreitz/requests/issues/3389
> > > > >
> > > > > For the source code release, do [unbundled] deps need to be
> cleared?
> > > From
> > > > > my understanding we only needed to clear the code we ship.
> > > > >
> > > > > If that's the case we've got work to do on the JS side.
> > > > >
> > > > > Max
> > > > >
> > > > >> On Wed, May 22, 2019 at 4:11 PM Bolke de Bruin <bdbruin@gmail.com
> >
> > > > wrote:
> > > > >>
> > > > >> Oef chardet is pulled in by requests. The usage of chardet (ie
> > > > triggered)
> > > > >> is unlikely as it is only used when the encoding is not set in
> > > headers.
> > > > >>
> > > > >> You could ask the maintainer of chardet to release under another
> > > > license.
> > > > >> This can be tough as their might be several people to contact that
> > > need
> > > > to
> > > > >> agree with relicensing. Or do a PR to make the usage of chardet
> > > > optional in
> > > > >> requests. Or use urllib and maybe create a wrapper that mimics
> > > requests.
> > > > >>
> > > > >> B.
> > > > >>
> > > > >>
> > > > >> Verstuurd vanaf mijn iPad
> > > > >>
> > > > >>> Op 23 mei 2019 om 00:03 heeft Alan Gates <al...@gmail.com>
> > het
> > > > >> volgende geschreven:
> > > > >>>
> > > > >>> +1 with caveats, see below. I looked at the LICENSE, NOTICE, and
> > > > >>> DISCLAIMER files, checked for any binary files (executables,
> > there's
> > > > >> plenty
> > > > >>> of image files in the distribution), and looked over the licenses
> > of
> > > > the
> > > > >>> dependencies.
> > > > >>>
> > > > >>> More information on the dependencies:
> > > > >>> I found https://pypi.org/project/pip-licenses/ which explains
> how
> > to
> > > > >> check
> > > > >>> licenses, very useful.
> > > > >>>
> > > > >>> The licenses of modules that will be pulled in when a system is
> > > > compiled
> > > > >> or
> > > > >>> run matter, as the system won't run without them. So it isn't ok
> > to
> > > > >> have a
> > > > >>> GPL licensed library that's necessarily pulled in at
> > compile/runtime,
> > > > as
> > > > >> to
> > > > >>> run the product you'll still be pulling in the GPL which will
> > > basically
> > > > >>> turn the whole thing GPL. (Optional or contrib components are
> > > > different,
> > > > >>> as users can choose not to run with them if they aren't ok with
> the
> > > > >> license
> > > > >>> of the optional component.)
> > > > >>>
> > > > >>> Running the above on the modules in setup.py, I see that the vast
> > > > >> majority
> > > > >>> are BSD, MIT, Apache, or PSFL, all of which are fine. The ones
> > that
> > > > >> aren't
> > > > >>> in that category are:
> > > > >>> certifi MPL-2.0: This is ok, as it's binary
> > > > >>> chardet LGPL Not Ok
> > > > >>> click UNKNOWN
> > > > >>> jsonschema UNKNOWN
> > > > >>> python-dateutil Dual License
> > > > >>> python-dotenv UNKNOWN
> > > > >>> python-geohash UNKNOWN
> > > > >>> python3-openid UNKNOWN
> > > > >>>
> > > > >>> The MPL one is fine since it's included in binary form. The
> > unknown
> > > > and
> > > > >>> dual license need some digging to determine what they are.
> > chardet,
> > > > the
> > > > >>> LGPL one, is not ok.
> > > > >>>
> > > > >>> Since this is an incubating release I am still voting +1, with
> the
> > > > caveat
> > > > >>> that the unknown licenses need to be figured out before the next
> > > > release,
> > > > >>> and the LGPL dependency will have to be removed. Right now I
> think
> > > > >> getting
> > > > >>> a release out is more important than fixing these issues.
> > > > >>>
> > > > >>> Alan.
> > > > >>>
> > > > >>> On Wed, May 22, 2019 at 2:01 PM Maxime Beauchemin <
> > > > >>> maximebeauchemin@gmail.com> wrote:
> > > > >>>
> > > > >>>> Oh actually the commands above just shows the dep tree.
> > > > >>>>
> > > > >>>> For deps in python there's
> > > > >>>> https://github.com/dhatim/python-license-check
> > > > >>>>
> > > > >>>> On the JS side I did some work here to attempt building the
> > LICENSE
> > > > file
> > > > >>>> dynamically as the dep tree evolves
> > > > >>>> https://github.com/apache/incubator-superset/pull/5801
> > > > >>>>
> > > > >>>> I thought validating the licenses of deps wasn't necessary for
> > > source
> > > > >>>> releases though. We may want to start the conversation on
> > > convenience
> > > > >>>> releases. To me having solid docker images (or just dockerfiles
> if
> > > > >> images
> > > > >>>> are troublesome) (that are lean and optimized to build fast)
> would
> > > be
> > > > >>>> ideal, especially if they are used in CI.
> > > > >>>>
> > > > >>>> Max
> > > > >>>>
> > > > >>>> On Wed, May 22, 2019 at 1:52 PM Maxime Beauchemin <
> > > > >>>> maximebeauchemin@gmail.com> wrote:
> > > > >>>>
> > > > >>>>> Python:
> > > > >>>>> pip install pipdeptree && pipdeptree
> > > > >>>>>
> > > > >>>>> NPM:
> > > > >>>>> cd superset/assets && npm ls
> > > > >>>>>
> > > > >>>>>> On Wed, May 22, 2019 at 11:09 AM Alan Gates <
> > alanfgates@gmail.com
> > > >
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Yes, I checked, it works now. I just haven't yet because I'm
> > > still
> > > > >>>>>> looking
> > > > >>>>>> at all the dependencies it pulls in. Maven makes this super
> > easy
> > > to
> > > > >> do,
> > > > >>>>>> but I need to learn enough about python setuptools to figure
> out
> > > how
> > > > >> to
> > > > >>>>>> check the licenses on those modules.
> > > > >>>>>>
> > > > >>>>>> Alan.
> > > > >>>>>>
> > > > >>>>>> On Wed, May 22, 2019 at 10:56 AM Bolke de Bruin <
> > > bdbruin@gmail.com>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Is the signature now verifiable? Otherwise it won’t pass the
> > IPMC
> > > > ...
> > > > >>>>>>>
> > > > >>>>>>> Verstuurd vanaf mijn iPad
> > > > >>>>>>>
> > > > >>>>>>>>> Op 22 mei 2019 om 19:26 heeft Maxime Beauchemin <
> > > > >>>>>>>> maximebeauchemin@gmail.com> het volgende geschreven:
> > > > >>>>>>>>
> > > > >>>>>>>> Oops, changing thread title this time around
> > > > >>>>>>>>
> > > > >>>>>>>> Vote passes!
> > > > >>>>>>>>
> > > > >>>>>>>> +3 binding votes (Max, Jeff & Abhishek)
> > > > >>>>>>>> +1 non-binding vote (Ville)
> > > > >>>>>>>>
> > > > >>>>>>>> No neutral or negative votes.
> > > > >>>>>>>>
> > > > >>>>>>>> On Tue, May 21, 2019 at 12:31 AM Jeff Feng
> > > > >>>>>> <jeff.feng@airbnb.com.invalid
> > > > >>>>>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> +1 binding
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Mon, May 20, 2019 at 3:54 PM Maxime Beauchemin <
> > > > >>>>>>>>> maximebeauchemin@gmail.com> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> @Alan, looks like I messed up the signature somehow. I got
> > > > tangled
> > > > >>>>>> into
> > > > >>>>>>>>>> adding a new entry (moving from my gmail to my apache.org
> > > > >>>> address),
> > > > >>>>>>>>>> deleting the old one and my svn kungfu is beyond rusty...
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Oh I think I just forgot to run "svn commit" (maybe i ran
> > "svn
> > > > >>>>>> update"
> > > > >>>>>>>>>> instead?), so you should just have to import that new KEYS
> > > file
> > > > >>>> and
> > > > >>>>>> it
> > > > >>>>>>>>>> should work.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Sorry about the confusion. All of this is pretty
> > error-prone,
> > > > >>>>>>> especially
> > > > >>>>>>>>>> the [few] first time[s] around.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Max
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Mon, May 20, 2019 at 11:29 AM Abhishek Sharma <
> > > > >>>>>>>>>> abhioncbr.apache@gmail.com>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> +1 binding.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Newly built docker image
> > > > >>>>>>>>>>> <
> > > > >>>>
> > > > >>
> > > >
> > >
> >
> https://cloud.docker.com/u/abhioncbr/repository/docker/abhioncbr/docker-superset
> > > > >>>>>>>>>>> working fine.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks
> > > > >>>>>>>>>>> Abhishek
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Mon, May 20, 2019 at 2:03 PM Alan Gates <
> > > > alanfgates@gmail.com
> > > > >>>>>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Max, when I check the signature (gpg --verify ) it tells
> > me:
> > > > >>>>>>>>>>>> gpg: Signature made Sat May 18 15:36:55 2019 PDT
> > > > >>>>>>>>>>>> gpg: using RSA key
> > > > >>>>>>>>>>> 8CA186C4568E92301E5F2491A3B3BE2CCC1BB7E4
> > > > >>>>>>>>>>>> gpg: Can't check signature: No public key
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I imported the KEYS file referenced in your message, but
> > it
> > > > >>>>>> doesn't
> > > > >>>>>>>>>>> appear
> > > > >>>>>>>>>>>> to contain that key. I think you need to either
> generate
> > a
> > > > new
> > > > >>>>>>>>>> signature
> > > > >>>>>>>>>>>> with the key in the file and upload that .asc file to
> the
> > > dist
> > > > >>>>>> site
> > > > >>>>>>>>> (no
> > > > >>>>>>>>>>>> need to rerole the release itself) or place the key you
> > used
> > > > >>>> into
> > > > >>>>>> the
> > > > >>>>>>>>>>> KEYS
> > > > >>>>>>>>>>>> file.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Alan.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Sat, May 18, 2019 at 4:01 PM Maxime Beauchemin <
> > > > >>>>>>>>>>>> maximebeauchemin@gmail.com> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Dear all,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> The source release 0.33.0 RC1 for Apache Superset is
> > baked
> > > > and
> > > > >>>>>>>>>>> available
> > > > >>>>>>>>>>>>> at:
> > > > >>>>>>>>>>>>>
> > https://dist.apache.org/repos/dist/dev/incubator/superset/
> > > ,
> > > > >>>>>> public
> > > > >>>>>>>>>>>>> keys are available
> > > > >>>>>>>>>>>>> at
> > > > >>>>
> > https://dist.apache.org/repos/dist/release/incubator/superset/KEYS
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> We're now attempting to use 0.33 as the base for the
> > first
> > > > >>>>>> release
> > > > >>>>>>>>> as
> > > > >>>>>>>>>>>>> opposed to 0.32 in previous attempts. Many
> > license-related
> > > > >>>> issues
> > > > >>>>>>>>> had
> > > > >>>>>>>>>>>> been
> > > > >>>>>>>>>>>>> solved by the process shipping visualizations as
> plugins,
> > > and
> > > > >>>>>> that
> > > > >>>>>>>>>>>>> migration wasn't completed on 0.32. This is the third
> ASF
> > > > >>>> release
> > > > >>>>>>>>>>>> candidate
> > > > >>>>>>>>>>>>> of Superset *We're still ironing out our release
> process,
> > > so
> > > > >>>>>> please
> > > > >>>>>>>>>>> bear
> > > > >>>>>>>>>>>>> with us and help if you can*.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> As I went along, I documented the process in
> > > > [yet-to-be-merged]
> > > > >>>>>>>>>>>>> RELEASING/README.md in the repo, latest edits here
> > > > >>>>>>>>>>>>> https://github.com/apache/incubator-superset/pull/7539
> .
> > > As
> > > > >>>> part
> > > > >>>>>>>>> of
> > > > >>>>>>>>>>>>> `RELEASING/`, we ship docker files to help package and
> > test
> > > > >>>>>>>>> releases.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> For context the `0.33` release branch was cut at SHA
> > > > 51068f007,
> > > > >>>>>>>>> that
> > > > >>>>>>>>>>> was
> > > > >>>>>>>>>>>>> merged on master on Apr 17th. From that common
> ancestor,
> > > the
> > > > >>>>>>>>>> following
> > > > >>>>>>>>>>>> list
> > > > >>>>>>>>>>>>> of commit was added as cherry-picks. The SHAs in the
> list
> > > > >>>> bellow
> > > > >>>>>>>>>>>> reference
> > > > >>>>>>>>>>>>> the cherries on the release branch, PR number are
> > available
> > > > to
> > > > >>>>>> get
> > > > >>>>>>>>>> more
> > > > >>>>>>>>>>>>> details.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> 7591a709 (tag: 0.33.0rc1, apache/release--0.33)
> 0.33.0rc1
> > > > >>>>>>>>>>>>> b7ffdb8b Improve instructions
> > > > >>>>>>>>>>>>> 4bbd68c6 Change babytux to open image in birth
> dashboard
> > > > >>>>>>>>>>>>> eaa679e8 remove unused LICENSE entries
> > > > >>>>>>>>>>>>> 42d50f9d Add Roboto font to LICENSE, remove glyphicons
> > > files
> > > > >>>>>>>>>>>>> 5ae2836b Address COPYRIGHT + LICENSE issues
> > > > >>>>>>>>>>>>> ea807f20 [WiP] Improvements related to ASF release
> > process
> > > > >>>>>>>>>>>>> c57ef5dc 0.31.0rc1.dev1
> > > > >>>>>>>>>>>>> 51068f00 Adding permission for
> > > can_only_access_owned_queries
> > > > >>>>>>>>> (#7234)
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> --
> > > > >>>>>>>>>
> > > > >>>>>>>>> *Jeff Feng*
> > > > >>>>>>>>> Product Lead
> > > > >>>>>>>>> m: (949)-610-5108
> > > > >>>>>>>>> twitter: @jtfeng
> > > > >>>>
> > > > >>
> > > >
> > >
> >
>