You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/03/02 23:33:30 UTC
[GitHub] [incubator-mxnet] josephevans opened a new pull request #19972: [WIP] [v1.x] Create tool for building source archives
josephevans opened a new pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972
## Description ##
This PR introduces a script that automates creating a release source archive and handles some licensing issues by removing some files that were identified to be non-compliant, and introducing a Apache RAT license check in the source generation process to ensure all Apache MXNet code is compliant.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586565102
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,82 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
+pushd $SRCDIR
+git submodule update --init --recursive
+echo "Checking out tag $MXNET_TAG..."
+git checkout $MXNET_TAG
+
+echo "Removing unwanted artifacts..."
+#### IMPORTANT ####
+# Remove artifacts which do not comply with the Apache Licensing Policy
+rm -rf R-package
Review comment:
Sure, any recommendations on where this file should live in the repo?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] Zha0q1 commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-789339396
@josephevans Does this same tool also work for master?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-789933178
I was going to remove `mx-theme` references in the LICENSE file, but it looks like @szha already has them removed in #19873. Should we also remove references to `mathjax` (from mkldnn docs) in LICENSE since we will no longer be shipping it out in the source archive? @leezu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha merged pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
szha merged pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-789300532
Hey @josephevans , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [centos-gpu, unix-gpu, edge, clang, windows-cpu, unix-cpu, centos-cpu, miscellaneous, windows-gpu, website, sanity]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586655643
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,86 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
Review comment:
I was assuming this script was going to be run manually when a new release is cut, do we really care about shedding a few seconds off here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586705085
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,86 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586654260
##########
File path: tools/source-exclude-artifacts.txt
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE:
+# This is a list of artifacts in the repository that should
+# not be included in source release archives due to licensing
+# restrictions.
+
+R-package
Review comment:
It was already excluded in the release runbook (see https://cwiki.apache.org/confluence/display/MXNET/Release+Process#ReleaseProcess-Step1.10.Createartifactsforthereleaseandpushtothedistfolder). Has this licensing issue been resolved? Can it be removed from the exclude list?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans edited a comment on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans edited a comment on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-790008656
> It would be great to add a check for all files listed in license and warn if any doesn't exist
Added this to the script. Found 2 items in the LICENSE file that are no longer in the source tree and removed them.
Perhaps this should be part of the normal sanity rat check CI pipeline so we catch the issues earlier instead of at release time. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586665084
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,86 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
+pushd $SRCDIR
+git submodule update --init --recursive
+echo "Checking out tag $MXNET_TAG..."
+git checkout $MXNET_TAG
+
+echo "Removing unwanted artifacts..."
+#### IMPORTANT ####
+# Remove artifacts which do not comply with the Apache Licensing Policy
+for d in $(cat tools/source-exclude-artifacts.txt | grep -v "^#"); do
+ if [[ -e $d ]]; then
+ echo "Removing $d from source archive..."
+ rm -rf $d
+ fi
+done
+
+# Remove other artifacts we do not want contained in the source archive
+rm -rf .DS_Store
+rm -rf CODEOWNERS
+find . -name ".git*" -print0 | xargs -0 rm -rf
Review comment:
Sure, but it won't remove `.github` (based on https://www.gnu.org/software/tar/manual/html_node/exclude.html) - I can also add that to the list on the line above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586652362
##########
File path: tools/source-exclude-artifacts.txt
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE:
+# This is a list of artifacts in the repository that should
+# not be included in source release archives due to licensing
+# restrictions.
+
+R-package
Review comment:
What is the concern about the R frontend?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586163489
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,82 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
+pushd $SRCDIR
+git submodule update --init --recursive
+echo "Checking out tag $MXNET_TAG..."
+git checkout $MXNET_TAG
+
+echo "Removing unwanted artifacts..."
+#### IMPORTANT ####
+# Remove artifacts which do not comply with the Apache Licensing Policy
+rm -rf R-package
Review comment:
Can you make this part of another file say another text file. so it is easy to audit for someone during voting without looking at a code file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans edited a comment on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans edited a comment on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-790008656
> It would be great to add a check for all files listed in license and warn if any doesn't exist
Added this to the script. Found 2 items in the LICENSE file that are no longer in the source tree and removed them.
Perhaps this should be part of the normal sanity rat check CI pipeline so we catch the issues earlier instead of at release time. WDYT? But that will have to be part of a different PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
szha commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-789982700
It would be great to add a check for all files listed in license and warn if any doesn't exist
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-789845891
> @josephevans Does this same tool also work for master?
There will be some slight modifications for master (there's no R-package in master, which we remove in v1.x.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-789942623
> Should we also remove references to mathjax (from mkldnn docs) in LICENSE since we will no longer be shipping it out in the source archive?
One idea is to automatically remove lines fro LICENSE that match a line in `tools/source-exclude-artifacts.txt`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r587909655
##########
File path: tools/source-exclude-artifacts.txt
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE:
+# This is a list of artifacts in the repository that should
+# not be included in source release archives due to licensing
+# restrictions.
+
+R-package
Review comment:
Verified with @szha, we still need to remove R-package, so this should be correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans removed a comment on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans removed a comment on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-789953759
> > Should we also remove references to mathjax (from mkldnn docs) in LICENSE since we will no longer be shipping it out in the source archive?
>
> One idea is to automatically remove lines fro LICENSE that match a line in `tools/source-exclude-artifacts.txt`?
Since it will no longer be distributed in the source archive, and it shouldn't be distributed in binary releases, why does it need to be in at all? I can definitely exclude lines that match our source-exclude-artifacts.txt list, but I'm not sure I understand the reasoning behind requiring it in the first place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586641898
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,82 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
+pushd $SRCDIR
+git submodule update --init --recursive
+echo "Checking out tag $MXNET_TAG..."
+git checkout $MXNET_TAG
+
+echo "Removing unwanted artifacts..."
+#### IMPORTANT ####
+# Remove artifacts which do not comply with the Apache Licensing Policy
+rm -rf R-package
Review comment:
Put them in `tools/source-exclude-artifacts.txt` for now, please let me know if there is a better location.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586774404
##########
File path: tools/source-exclude-artifacts.txt
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE:
+# This is a list of artifacts in the repository that should
+# not be included in source release archives due to licensing
+# restrictions.
+
+R-package
Review comment:
I don't know why it was listed there. That is very dubious, because in that case R-package has never been an official feature of MXNet and we may just delete the code directly as it's no longer maintained anyways..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-790811494
> > Should we also remove references to mathjax (from mkldnn docs) in LICENSE since we will no longer be shipping it out in the source archive?
>
> One idea is to automatically remove lines fro LICENSE that match a line in `tools/source-exclude-artifacts.txt`?
OK, I updated to remove lines in LICENSE for stuff we remove the source archive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-790008656
> It would be great to add a check for all files listed in license and warn if any doesn't exist
Added this to the script. Found 2 items in the LICENSE file that are no longer in the source tree and removed them.,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586655643
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,86 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
Review comment:
I was assuming this script was going to be run manually when a new release is cut, do we really care about shedding a few seconds off here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586651635
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,86 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
+pushd $SRCDIR
+git submodule update --init --recursive
+echo "Checking out tag $MXNET_TAG..."
+git checkout $MXNET_TAG
+
+echo "Removing unwanted artifacts..."
+#### IMPORTANT ####
+# Remove artifacts which do not comply with the Apache Licensing Policy
+for d in $(cat tools/source-exclude-artifacts.txt | grep -v "^#"); do
+ if [[ -e $d ]]; then
+ echo "Removing $d from source archive..."
+ rm -rf $d
+ fi
+done
+
+# Remove other artifacts we do not want contained in the source archive
+rm -rf .DS_Store
+rm -rf CODEOWNERS
+find . -name ".git*" -print0 | xargs -0 rm -rf
Review comment:
How about passing `--exclude-vcs` to `tar` instead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
josephevans commented on pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#issuecomment-789953759
> > Should we also remove references to mathjax (from mkldnn docs) in LICENSE since we will no longer be shipping it out in the source archive?
>
> One idea is to automatically remove lines fro LICENSE that match a line in `tools/source-exclude-artifacts.txt`?
Since it will no longer be distributed in the source archive, and it shouldn't be distributed in binary releases, why does it need to be in at all? I can definitely exclude lines that match our source-exclude-artifacts.txt list, but I'm not sure I understand the reasoning behind requiring it in the first place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #19972: [WIP] [v1.x] Create tool for building source archives
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #19972:
URL: https://github.com/apache/incubator-mxnet/pull/19972#discussion_r586650779
##########
File path: tools/create_source_archive.sh
##########
@@ -0,0 +1,86 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -e
+
+MXNET_BRANCH=$1
+MXNET_TAG=$2
+CMD=$(basename $0)
+
+if [[ -z "$MXNET_BRANCH" || -z "$MXNET_TAG" ]]; then
+ echo "Usage: $CMD <branch> <tag>"
+ echo " where <branch> is the branch the tag was cut from."
+ echo " <tag> is the tag you want to build a release for."
+ echo ""
+ echo " example: $CMD v1.8.x 1.8.0.rc3"
+ exit -1
+fi
+
+TAR=tar
+if [[ $(uname) == "Darwin" ]]; then
+ TAR=gtar
+fi
+
+# make sure gnu tar is installed
+which $TAR > /dev/null
+if [[ $? -ne 0 ]]; then
+ echo "It looks like you don't have GNU tar installed."
+ echo ""
+ echo "For OSX users, please install gnu-tar using the command 'brew install gnu-tar'"
+ exit -1
+fi
+
+SRCDIR=apache-mxnet-src-$MXNET_TAG-incubating
+TARBALL=$SRCDIR.tar.gz
+
+# clone the repo and checkout the tag
+echo "Cloning the MXNet repository..."
+git clone -b $MXNET_BRANCH https://github.com/apache/incubator-mxnet.git $SRCDIR
Review comment:
Please use `--depth 1 --recurse-submodules --shallow-submodules` to speed up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org