You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2017/03/26 21:20:21 UTC
[GitHub] spark pull request #17437: [SPARK-20102] Fix nightly snapshot and RC packagi...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/17437
[SPARK-20102] Fix nightly snapshot and RC packaging scripts w/ two minor build fixes
## What changes were proposed in this pull request?
The master snapshot publisher builds are currently broken due to two minor build issues:
1. For unknown reasons, the LFTP `mkdir -p` command began throwing errors when the remote FTP directory already exists. This change of behavior might have been caused by configuration changes in the ASF's SFTP server, but I'm not entirely sure of that. To work around this problem, this patch updates the script to ignore errors from the `lftp mkdir -p` commands.
2. The PySpark `setup.py` file references a non-existent `pyspark.ml.stat` module, causing Python packaging to fail by complaining about a missing directory. The fix is to simply drop that line from the setup script.
## How was this patch tested?
The LFTP fix was tested by manually running the failing commands on AMPLab Jenkins against the ASF SFTP server. The PySpark fix was tested locally.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark spark-20102
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17437.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17437
----
commit 213fd2f12bf39dde8311e02e01216af9430c3eb0
Author: Josh Rosen <jo...@databricks.com>
Date: 2017-03-23T20:31:11Z
Fix LFTP mkdir -p
commit 8d29b5aff5ee45fa54e90d96685d680fc18e9313
Author: Josh Rosen <jo...@databricks.com>
Date: 2017-03-24T00:13:48Z
Remove -f
commit 867ca2ccab8fe75861a246e166da175086c45248
Author: Josh Rosen <jo...@databricks.com>
Date: 2017-03-24T20:14:07Z
Remove pyspark.ml.stat (non-existent?) from setup.py file.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17437: [SPARK-20102] Fix nightly packaging and RC packaging scr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17437
**[Test build #75246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75246/testReport)** for PR 17437 at commit [`867ca2c`](https://github.com/apache/spark/commit/867ca2ccab8fe75861a246e166da175086c45248).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17437: [SPARK-20102] Fix nightly packaging and RC packag...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17437
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17437: [SPARK-20102] Fix nightly packaging and RC packaging scr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17437
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75246/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17437: [SPARK-20102] Fix nightly packaging and RC packaging scr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17437
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17437: [SPARK-20102] Fix nightly snapshot and RC packaging scri...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17437
**[Test build #75246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75246/testReport)** for PR 17437 at commit [`867ca2c`](https://github.com/apache/spark/commit/867ca2ccab8fe75861a246e166da175086c45248).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17437: [SPARK-20102] Fix nightly snapshot and RC packaging scri...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/17437
/cc @marmbrus @holdenk, this patch should fix the build issues blocking Spark 2.1.1 RC1 builds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17437: [SPARK-20102] Fix nightly packaging and RC packaging scr...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/17437
I'm going to merge this into master and branch-2.1 then will manually kick off the packaging jobs so that we can get back to publishing nightly package snapshots.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17437: [SPARK-20102] Fix nightly packaging and RC packag...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/17437#discussion_r108078826
--- Diff: python/setup.py ---
@@ -167,7 +167,6 @@ def _supports_symlinks():
'pyspark.ml',
'pyspark.ml.linalg',
'pyspark.ml.param',
- 'pyspark.ml.stat',
--- End diff --
I think it was. There is `pyspark.mllib.stat` but no `pyspark.ml.stat`. Seems it is mistakenly missing the non-parity between `mllib` and `ml` sub-modules.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17437: [SPARK-20102] Fix nightly snapshot and RC packagi...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17437#discussion_r108071973
--- Diff: python/setup.py ---
@@ -167,7 +167,6 @@ def _supports_symlinks():
'pyspark.ml',
'pyspark.ml.linalg',
'pyspark.ml.param',
- 'pyspark.ml.stat',
--- End diff --
@holdenk @viirya, quick question here: I see that this `pyspark.ml.stat` line was added in #16465. The problem with this line is that it seems to cause failures during `python setup.py sdist` packaging because that process searches for a non-existent `pyspark/ml/stat` file / directory.
Was this `ml.stat` just a typo?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org