You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by "Ting(Goden) Yao" <ty...@pivotal.io> on 2015/12/18 20:04:34 UTC
Re: Propose an Apache HAWQ Release
I just tagged "2.0.0-beta-incubating-RC2" based on latest commit:
https://github.com/apache/incubator-hawq/commit/b6cb39c37e83fb5ee542b9f953c3a5e28e4617eb
Please mentors review and provide feedback.
-Goden
On Tue, Nov 24, 2015 at 10:09 AM Ting(Goden) Yao <ty...@pivotal.io> wrote:
> Just correct the typos : they are HAWQ-183, HAWQ-184, HAWQ-185 and
> HAWQ-186 to track all the new issues Roman discovered for
> 2.0.0-beta-incubating release.
> I've updated release page with the latest info.
>
> On Mon, Nov 23, 2015 at 11:57 PM Lei Chang <ch...@gmail.com> wrote:
>
>> Good catch. HAWQ-1863 HAWQ-1864 HAWQ-1865 HAWQ-1866 are used to track
>> this.
>>
>> Cheers
>> Lei
>>
>>
>> On Tue, Nov 24, 2015 at 12:37 PM, Roman Shaposhnik <ro...@shaposhnik.org>
>> wrote:
>>
>> > On Mon, Nov 23, 2015 at 12:45 PM, Ting(Goden) Yao <ty...@pivotal.io>
>> wrote:
>> > > Yes I meant "RC1" , missed typing it in the previous email.
>> >
>> > Great! My cursory look at the tag produced the following:
>> > 1. We need to add the DISCLAIMER file
>> > 2. There are a few things checked into the source tree that
>> > definitely shouldn't be in the release tarball:
>> > * anythings that's *.tar.gz
>> > * anything that's a dependency jar
>> > * ./depends/libyarn/releng/bin/templates/.core-site-secure-ha.xml.swp
>> > * ./pxf/gradle/wrapper/gradle-wrapper.jar
>> > * ./contrib/hawq-hadoop/hawq-mapreduce-tool/lib/*.jar
>> > * ./src/test/regress/data/pxf/*.class
>> > * ./src/test/regress/data/pxf/*.class
>> > * ./tools/bin/ext/yaml/*.pyc
>> >
>> > 3. There are a few things that I'd like to understand
>> > why they are there:
>> > * ./depends/libyarn/releng/make/dependencies/cacerts
>> >
>> > Finally (and somewhat surprisingly) the RAT check turned out
>> > to be pretty alarming. I thought that you guys took care of
>> > the licensing headers some time ago. Am I misremembering?
>> >
>> > Anyway, here's how to reproduce and marvel at "5030 Unknown Licenses".
>> >
>> > $ java -jar ~/src/apache/rat-0.11/apache-rat-0.11.jar .
>> >
>> > *****************************************************
>> > Summary
>> > -------
>> > Generated at: 2015-11-23T19:40:00-08:00
>> > Notes: 79
>> > Binaries: 789
>> > Archives: 20
>> > Standards: 5871
>> >
>> > Apache Licensed: 825
>> > Generated Documents: 0
>> >
>> > JavaDocs are generated and so license header is optional
>> > Generated files do not required license headers
>> >
>> > 5030 Unknown Licenses
>> >
>> > *******************************
>> >
>>
>
Re: Propose an Apache HAWQ Release
Posted by Konstantin Boudnik <co...@apache.org>.
On Wed, Dec 23, 2015 at 05:03PM, Roman Shaposhnik wrote:
> On Wed, Dec 23, 2015 at 4:56 PM, Noa Horn <nh...@pivotal.io> wrote:
> > Regarding 2 (gradle), thank you for the reference. I will open a JIRA for
> > it.
>
> Great!
>
> > Do we have to credit bigtop or can we just take the file as is?
>
> A line in NOTICE would be nice ;-)
As long as your copy stays under ASL2 terms - you're good ;)
Cos
Re: Propose an Apache HAWQ Release
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Wed, Dec 23, 2015 at 10:21 PM, Wen Lin <wl...@pivotal.io> wrote:
> Hi, Roman,
>
> For these lines, I can make it more specific to some subtree.
> <exclude>**/*.mk</exclude>
> <exclude>**/*.mak</exclude>
> <exclude>**/*.yaml</exclude>
> <exclude>**/*.m4</exclude>
> <exclude>config/*</exclude>
> <exclude>**/*.rc</exclude>
> <exclude>**/*.f</exclude>
> <exclude>**/Makefile*</exclude>
> For the other files, if it is too long to read, what do you suggest make it
> simple?
> For example, in src/backend/executor, some files belong to Postgres, some
> files belong to Apache in same directory.
>
> It is hard to write a wildcard expression which can exclude Postgres files,
> unless Iist all the files belong to Postgres in exclude file.
I've been thinking that perhaps we can automate the PG-derived
code detection process by including something like the snippet
bellow in pom.xml.
That'll give us part of the solution. Some of the files that were derived
from PG now carry Greenplum, EMC and Pivotal copyrights. For those
files an explicit ALv2 header will be required.
> There are a lot of similar situations like this. That's why the exclude
> list is very long.
> Do you have any idea for handling this?
Personally, I'd like to leverage RAT as much as possible here for
actually detecting licenses, not just excluding files. I know my
suggestion below may not be bullet proof, so I'd love to know
what other mentors think about it.
Thanks,
Roman.
RAT configuration snippet:
<licenses>
<license
implementation="org.apache.rat.analysis.license.SimplePatternBasedLicense">
<licenseFamilyCategory>PGSQL</licenseFamilyCategory>
<licenseFamilyName>PostgreSQL License</licenseFamilyName>
<notes></notes>
<patterns>
<pattern>$PostgreSQL:</pattern>
<pattern>PostgreSQL Global Development Group</pattern>
</patterns>
</license>
</licenses>
Re: Propose an Apache HAWQ Release
Posted by Wen Lin <wl...@pivotal.io>.
Hi, Roman,
For these lines, I can make it more specific to some subtree.
<exclude>**/*.mk</exclude>
<exclude>**/*.mak</exclude>
<exclude>**/*.yaml</exclude>
<exclude>**/*.m4</exclude>
<exclude>config/*</exclude>
<exclude>**/*.rc</exclude>
<exclude>**/*.f</exclude>
<exclude>**/Makefile*</exclude>
For the other files, if it is too long to read, what do you suggest make it
simple?
For example, in src/backend/executor, some files belong to Postgres, some
files belong to Apache in same directory.
It is hard to write a wildcard expression which can exclude Postgres files,
unless Iist all the files belong to Postgres in exclude file.
There are a lot of similar situations like this. That's why the exclude
list is very long.
Do you have any idea for handling this?
Thanks and Regards!
On Thu, Dec 24, 2015 at 12:09 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Wed, Dec 23, 2015 at 6:41 PM, Radar Da lei <rl...@pivotal.io> wrote:
> > Hi Roman,
> >
> > For item 0 and 1, I'm working on this, and there is a jira for it:
> > https://issues.apache.org/jira/browse/HAWQ-271
> >
> > These python source code will be removed, and user need to install them
> > before install hawq, maybe by using python pip install.
>
> Great! Seems like you guys are on a trajectory to take care of all the
> feedback I provided. I suggest once that is done you simply follow
> up with a more formal VOTE for RC3 as opposed to just doing the
> tagging.
>
> Btw, if may be a good idea to document the release steps on your
> wiki similar to how we did it for, lets say, Bigtop:
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
>
> Thanks,
> Roman.
>
Re: Propose an Apache HAWQ Release
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Wed, Dec 23, 2015 at 6:41 PM, Radar Da lei <rl...@pivotal.io> wrote:
> Hi Roman,
>
> For item 0 and 1, I'm working on this, and there is a jira for it:
> https://issues.apache.org/jira/browse/HAWQ-271
>
> These python source code will be removed, and user need to install them
> before install hawq, maybe by using python pip install.
Great! Seems like you guys are on a trajectory to take care of all the
feedback I provided. I suggest once that is done you simply follow
up with a more formal VOTE for RC3 as opposed to just doing the
tagging.
Btw, if may be a good idea to document the release steps on your
wiki similar to how we did it for, lets say, Bigtop:
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
Thanks,
Roman.
Re: Propose an Apache HAWQ Release
Posted by Radar Da lei <rl...@pivotal.io>.
Hi Roman,
For item 0 and 1, I'm working on this, and there is a jira for it:
https://issues.apache.org/jira/browse/HAWQ-271
These python source code will be removed, and user need to install them
before install hawq, maybe by using python pip install.
Thanks.
Regards,
Radar
On Thu, Dec 24, 2015 at 9:03 AM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Wed, Dec 23, 2015 at 4:56 PM, Noa Horn <nh...@pivotal.io> wrote:
> > Regarding 2 (gradle), thank you for the reference. I will open a JIRA for
> > it.
>
> Great!
>
> > Do we have to credit bigtop or can we just take the file as is?
>
> A line in NOTICE would be nice ;-)
>
> Thanks,
> Roman.
>
Re: Propose an Apache HAWQ Release
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Wed, Dec 23, 2015 at 4:56 PM, Noa Horn <nh...@pivotal.io> wrote:
> Regarding 2 (gradle), thank you for the reference. I will open a JIRA for
> it.
Great!
> Do we have to credit bigtop or can we just take the file as is?
A line in NOTICE would be nice ;-)
Thanks,
Roman.
Re: Propose an Apache HAWQ Release
Posted by Noa Horn <nh...@pivotal.io>.
Regarding 2 (gradle), thank you for the reference. I will open a JIRA for
it.
Do we have to credit bigtop or can we just take the file as is?
On Wed, Dec 23, 2015 at 4:39 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:
> On Fri, Dec 18, 2015 at 11:04 AM, Ting(Goden) Yao <ty...@pivotal.io> wrote:
> > I just tagged "2.0.0-beta-incubating-RC2" based on latest commit:
> >
> https://github.com/apache/incubator-hawq/commit/b6cb39c37e83fb5ee542b9f953c3a5e28e4617eb
>
> This looks *much* better than RC1. Great progress, team!
>
> Still a few things are left that need to be addressed:
> 0. Content of tools/bin/pythonSrc doesn't look good to me.
> There's GPL stuff in there that absolutely MUST be purged
> from our repo. There are a few things under MIT and BSD
> licenses in there as well, but since you will have to come up
> with a solution for GPL archives you may as well get rid of
> all of them.
>
> 1. netperf-2.4.3.tar.gz needs to be removed. It is under a specific
> license from HP and we can't carry it in the tarball release of HAWQ
>
> 2. I still think that we need to get rid of Gradle jar. In order to
> solve
> boostraping problem you can either use the approach Bigtop is taking:
> https://github.com/apache/bigtop/blob/master/gradlew#L18
> (just replace your gradelw with Bigtop's). Or you can mandate Gradle.
>
> 3. I really appreciate RAT integration, but looking at the exclude list
> I don't think the following belongs there:
> <exclude>**/*.mk</exclude>
> <exclude>**/*.mak</exclude>
> <exclude>**/*.yaml</exclude>
> <exclude>**/*.m4</exclude>
> <exclude>config/*</exclude>
> <exclude>**/*.rc</exclude>
> <exclude>**/*.f</exclude>
> <exclude>**/Makefile*</exclude>
> Instead of wholesale exclusion like that I'd like to see exclusion
> based on
> subtrees of the source tree. IOW, if a file that matches one of the
> above
> masks comes from PG just mark it so and mark a folder. Excluding a whole
> class of files based on extension makes RAT less useful in the future.
>
> 4. In general, I'd like to see the exclude list for RAT be more
> structured
> and concise. It is currently 446 entries long and pretty much impossible
> to review properly.
>
> Thanks,
> Roman.
>
Re: Propose an Apache HAWQ Release
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Fri, Dec 18, 2015 at 11:04 AM, Ting(Goden) Yao <ty...@pivotal.io> wrote:
> I just tagged "2.0.0-beta-incubating-RC2" based on latest commit:
> https://github.com/apache/incubator-hawq/commit/b6cb39c37e83fb5ee542b9f953c3a5e28e4617eb
This looks *much* better than RC1. Great progress, team!
Still a few things are left that need to be addressed:
0. Content of tools/bin/pythonSrc doesn't look good to me.
There's GPL stuff in there that absolutely MUST be purged
from our repo. There are a few things under MIT and BSD
licenses in there as well, but since you will have to come up
with a solution for GPL archives you may as well get rid of
all of them.
1. netperf-2.4.3.tar.gz needs to be removed. It is under a specific
license from HP and we can't carry it in the tarball release of HAWQ
2. I still think that we need to get rid of Gradle jar. In order to solve
boostraping problem you can either use the approach Bigtop is taking:
https://github.com/apache/bigtop/blob/master/gradlew#L18
(just replace your gradelw with Bigtop's). Or you can mandate Gradle.
3. I really appreciate RAT integration, but looking at the exclude list
I don't think the following belongs there:
<exclude>**/*.mk</exclude>
<exclude>**/*.mak</exclude>
<exclude>**/*.yaml</exclude>
<exclude>**/*.m4</exclude>
<exclude>config/*</exclude>
<exclude>**/*.rc</exclude>
<exclude>**/*.f</exclude>
<exclude>**/Makefile*</exclude>
Instead of wholesale exclusion like that I'd like to see exclusion based on
subtrees of the source tree. IOW, if a file that matches one of the above
masks comes from PG just mark it so and mark a folder. Excluding a whole
class of files based on extension makes RAT less useful in the future.
4. In general, I'd like to see the exclude list for RAT be more structured
and concise. It is currently 446 entries long and pretty much impossible
to review properly.
Thanks,
Roman.