You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2016/01/04 00:50:53 UTC
[GitHub] spark pull request: Add missing Hadoop profiles to dev/run-tests-*...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/10565
Add missing Hadoop profiles to dev/run-tests-*.py scripts
There are a couple of places in the `dev/run-tests-*.py` scripts which deal with Hadoop profiles, but the set of profiles that they handle does not include all Hadoop profiles defined in our POM. This patch updates those scripts to achieve completeness.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark add-missing-hadoop-profiles-in-test-scripts
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10565.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10565
----
commit 426c3e2fdea7e9a40a3149974a3f9b9912a0bb8e
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-01-03T23:47:41Z
Add missing Hadoop profiles in test scripts.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168557245
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168557719
```
[info] spark-sql: found 2 potential binary incompatibilities (filtered 527)
[error] * method leafFiles()scala.collection.mutable.Map in class org.apache.spark.sql.sources.HadoopFsRelation#FileStatusCache has now a different result type; was: scala.collection.mutable.Map, is now: scala.collection.mutable.LinkedHashMap
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.sources.HadoopFsRelation#FileStatusCache.leafFiles")
[error] * method leafFiles_=(scala.collection.mutable.Map)Unit in class org.apache.spark.sql.sources.HadoopFsRelation#FileStatusCache's type has changed; was (scala.collection.mutable.Map)Unit, is now: (scala.collection.mutable.LinkedHashMap)Unit
[error] filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.sources.HadoopFsRelation#FileStatusCache.leafFiles_=")
[warn] /home/jenkins/workspace/SparkPullRequestBuilder/mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala:388: method
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168558078
It occurred to me that we should probably also add these profiles to the `dev/deps` lists as well, esp. since the `hadoop-2.6` profile has some dependency overrides. I might as well do that as part of this PR to collect all of the Hadoop profile parity changes in one place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168557882
**[Test build #48620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48620/consoleFull)** for PR 10565 at commit [`426c3e2`](https://github.com/apache/spark/commit/426c3e2fdea7e9a40a3149974a3f9b9912a0bb8e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.4] A...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168556598
**[Test build #48619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48619/consoleFull)** for PR 10565 at commit [`426c3e2`](https://github.com/apache/spark/commit/426c3e2fdea7e9a40a3149974a3f9b9912a0bb8e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.4] A...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10565#discussion_r48699087
--- Diff: dev/run-tests.py ---
@@ -295,13 +295,14 @@ def exec_sbt(sbt_args=()):
def get_hadoop_profiles(hadoop_version):
"""
- For the given Hadoop version tag, return a list of SBT profile flags for
+ For the given Hadoop version tag, return a list of Maven/SBT profile flags for
--- End diff --
These flags are used in both Maven and SBT, hence this clarification (to see this, grep the file for usages of `get_hadoop_profiles`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168564758
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48620/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168566346
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48621/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.4] A...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168557143
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48619/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168564702
**[Test build #48620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48620/consoleFull)** for PR 10565 at commit [`426c3e2`](https://github.com/apache/spark/commit/426c3e2fdea7e9a40a3149974a3f9b9912a0bb8e).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168564757
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168560673
**[Test build #48621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48621/consoleFull)** for PR 10565 at commit [`08c2127`](https://github.com/apache/spark/commit/08c21270c0ffb2a0262676a8af22aa1eb365e4d5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.4] A...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168557142
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168557689
This MiMa failure looks interesting; I wonder if we had a non-explicit return type which is affected by the Hadoop version...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.4] A...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10565#discussion_r48699084
--- Diff: dev/run-tests.py ---
@@ -295,13 +295,14 @@ def exec_sbt(sbt_args=()):
def get_hadoop_profiles(hadoop_version):
"""
- For the given Hadoop version tag, return a list of SBT profile flags for
+ For the given Hadoop version tag, return a list of Maven/SBT profile flags for
building and testing against that Hadoop version.
"""
sbt_maven_hadoop_profiles = {
"hadoop2.2": ["-Pyarn", "-Phadoop-2.2"],
- "hadoop2.3": ["-Pyarn", "-Phadoop-2.3", "-Dhadoop.version=2.3.0"],
--- End diff --
Setting `hadoop.version` is unnecessary here; it's already set in the profile.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168590060
The test failures are unrelated, so I'm going to merge this into master in order to unblock some other fixes to test infra.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168566344
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.4] A...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168557133
**[Test build #48619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48619/consoleFull)** for PR 10565 at commit [`426c3e2`](https://github.com/apache/spark/commit/426c3e2fdea7e9a40a3149974a3f9b9912a0bb8e).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.4] A...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168556248
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA] Add missing Hadoo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10565
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12612][PROJECT-INFRA][test-hadoop2.3] A...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10565#issuecomment-168566300
**[Test build #48621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48621/consoleFull)** for PR 10565 at commit [`08c2127`](https://github.com/apache/spark/commit/08c21270c0ffb2a0262676a8af22aa1eb365e4d5).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org