You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2016/04/12 23:04:14 UTC
[GitHub] spark pull request: [SPARK-14574][BUILD] Stop cross-building pure ...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/12334
[SPARK-14574][BUILD] Stop cross-building pure Java modules; simplify build target dirs
Spark has a few modules which do not depend on Scala, including `launcher`, `unsafe`, `sketch`, and the `network-*` libraries. However, we currently cross build and publish these artifacts for different Scala versions.
This patch refactors Spark's build so that pure-Java modules can be published once and so that their artifact ids do not include `_2.xx` suffixes. A consequence of this change is that pure Java modules' build output directories will be `/target/classes` rather than `/target/scala-*/classes`. For uniformity, I updated _all_ projects, including Scala ones, to use this output directory convention. I think that the only reason to prefer SBT's `crossPaths` convention is if you're using SBT's cross-build support (which, AFAIK, doesn't currently work for Spark and isn't something that we're aiming to support).
The `target` directory changes, in turn, allowed me to significantly simplify the launcher code and removed the need to detect and set `SPARK_SCALA_VERSION` in the launcher scripts. This simplification will have a huge payoff when if we eventually have to simultaneously support Scala 2.10.x, 2.11.x, and 2.12.x, since there's now much less version detection logic to maintain.
/cc @rxin @marmbrus @ahirreddy @srowen @vanzin
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark remove-unnecessary-scala-exposure-in-poms
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12334.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12334
----
commit 42947032e565f6eccfa2ccceb26b00a3da7d9ad6
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-12T19:02:46Z
Remove Scala version from parent POM artifact ID
commit 34d52d3b66cc9f50e0d5620cba37580716848bed
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-12T20:11:46Z
WIP towards fixing artifact names and build output directories.
commit c7db9acd49179859cd8b1083923150e63bff5676
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-12T20:22:01Z
Remove SPARK_SCALA_VERSION env. var. and clean up launcher code.
commit f4de68930339227bf414d4a27cbff9222fee322e
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-12T20:37:33Z
Remove more references to target/scala-
commit 04161558a1097b634d459117dac66bd661b7cb79
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-12T20:42:56Z
More updates in examples/
commit 449e93464b16b97c8af3806b73d96059cba5b45c
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-12T20:49:11Z
Remove Scala download from build/mvn
commit acd738f33c1210bfae44d631c86fbd830a95d7d1
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-12T20:53:11Z
Update spark-sketch artifactid and pom.
commit 944e8ab3da77a406fec9a8c1da118299f228a3e0
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-12T20:53:53Z
More cleanup in make-distribution.sh
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209200573
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55668/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59610525
--- Diff: assembly/pom.xml ---
@@ -20,7 +20,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
- <artifactId>spark-parent_2.11</artifactId>
+ <artifactId>spark-parent</artifactId>
--- End diff --
Based on some of the comments in https://issues.apache.org/jira/browse/MNG-2971, https://issues.apache.org/jira/browse/MNG-4223, and other tickets I believe that the intent is that properties should be expanded when installing / deploying POMs.
It looks like a somewhat more explicit way of ensuring this expansion would be to use the http://www.mojohaus.org/flatten-maven-plugin/ plugin.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209114379
**[Test build #55650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55650/consoleFull)** for PR 12334 at commit [`944e8ab`](https://github.com/apache/spark/commit/944e8ab3da77a406fec9a8c1da118299f228a3e0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209154693
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209616832
> regarding not supporting cross builds for Spark, is that really something we don't want to support? I've hacked together something (non-Spark-related) in maven to allow cross-compiling in the same build, and I think that if we're going to support multiple versions of Scala, that might be a good thing to have in Spark; it's not unusual for the 2.10 build to be broken these days because it's not exercised by the PR builders.
Whether we compile for multiple versions of Scala as part of the same CI build does not necessarily imply usage of [SBT's cross-build](http://www.scala-sbt.org/0.13/docs/Cross-Build.html) feature (e.g. being able to do "sbt + package" to build for all Scala versions at the same time. I think that it's going to be prohibitively difficult to support the use of SBT's `+` operator given our use of the POM reader for declaring dependencies. I guess the TL;DR is that I think the best way to build for multiple Scala versions in CI is to just kick off multiple CI builds in parallel or have our bash / Python CI scripts orchestrate that at a higher level rather than trying to push that logic into Maven or SBT so that it can be done in a single build-tool invocation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59542335
--- Diff: pom.xml ---
@@ -274,6 +274,11 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.esotericsoftware</groupId>
--- End diff --
Also indented a little too far. Is updating the kryo version part of this change too?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59541871
--- Diff: assembly/pom.xml ---
@@ -20,7 +20,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
- <artifactId>spark-parent_2.11</artifactId>
+ <artifactId>spark-parent</artifactId>
--- End diff --
Interesting, so parent is no longer Scala version specific? I would have thought it might be as it provides vars for the Scala version. I expect you've thought this through more than I have though
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209834673
> I think the best way to build for multiple Scala versions in CI is to just kick off multiple CI builds in parallel or have our bash / Python CI scripts orchestrate that
That's doable, but my main gripe with the current way cross-compilation is done is that it requires you to pollute your working file set. You can't just say "build for Scala 2.10" with just command line options; you have to change all the poms.
There are ways to reduce the annoyance (e.g. commit the pom changes and later remember to remove them from the branch), but if the build infrastructure handled that more cleanly, it would be great.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD] Stop cross-building pure ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209102663
**[Test build #55647 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55647/consoleFull)** for PR 12334 at commit [`944e8ab`](https://github.com/apache/spark/commit/944e8ab3da77a406fec9a8c1da118299f228a3e0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209422826
That cleanup sure is nice. Off the top of my head I don't know of anything else that needs changing, but we'll see how this flies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59598076
--- Diff: assembly/pom.xml ---
@@ -20,7 +20,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
- <artifactId>spark-parent_2.11</artifactId>
+ <artifactId>spark-parent</artifactId>
--- End diff --
Empirically, it looks like the generated POMS which are used for publishing end up with all variables in the `<dependencies>` section having been expanded according to whichever profile was active at publication time. For example, in http://central.maven.org/maven2/org/apache/spark/spark-core_2.10/1.6.1/spark-core_2.10-1.6.1.pom, there are no references to `scala.binary.version` in the `<dependencies>` section. There _are_ references in the `<build>` section but I don't think that this is relevant to consumers of this POM since I don't expect people to depend on our child POMs in their own builds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209200262
**[Test build #55668 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55668/consoleFull)** for PR 12334 at commit [`944e8ab`](https://github.com/apache/spark/commit/944e8ab3da77a406fec9a8c1da118299f228a3e0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59542067
--- Diff: common/unsafe/pom.xml ---
@@ -37,8 +37,8 @@
<dependencies>
<dependency>
- <groupId>com.twitter</groupId>
- <artifactId>chill_${scala.binary.version}</artifactId>
+ <groupId>com.esotericsoftware</groupId>
--- End diff --
Truly small nit but these got indented too far
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209112790
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59542489
--- Diff: yarn/src/main/scala/org/apache/spark/launcher/YarnCommandBuilderUtils.scala ---
@@ -31,11 +30,7 @@ private[spark] object YarnCommandBuilderUtils {
}
def findJarsDir(sparkHome: String): String = {
- val scalaVer = Properties.versionNumberString
- .split("\\.")
- .take(2)
- .mkString(".")
- CommandBuilderUtils.findJarsDir(sparkHome, scalaVer, true)
+ CommandBuilderUtils.findJarsDir(sparkHome, true)
--- End diff --
Can we just get rid of this class if it just calls `CommandBuilderUtils`? I feel like I might be forgetting a reason it exists.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59543207
--- Diff: yarn/src/main/scala/org/apache/spark/launcher/YarnCommandBuilderUtils.scala ---
@@ -31,11 +30,7 @@ private[spark] object YarnCommandBuilderUtils {
}
def findJarsDir(sparkHome: String): String = {
- val scalaVer = Properties.versionNumberString
- .split("\\.")
- .take(2)
- .mkString(".")
- CommandBuilderUtils.findJarsDir(sparkHome, scalaVer, true)
+ CommandBuilderUtils.findJarsDir(sparkHome, true)
--- End diff --
`CommandBuilderUtils` is package-private and can't be accessed by the yarn module's classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209154695
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55650/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209148425
**[Test build #55647 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55647/consoleFull)** for PR 12334 at commit [`944e8ab`](https://github.com/apache/spark/commit/944e8ab3da77a406fec9a8c1da118299f228a3e0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209656065
Going to close this for now as "later", since it's going to be prohibitively difficult to ensure that we continue to publish flattened poms and there's some measure of risk here given how parts of this (and the current build) appear to work by accident.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209156854
**[Test build #55668 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55668/consoleFull)** for PR 12334 at commit [`944e8ab`](https://github.com/apache/spark/commit/944e8ab3da77a406fec9a8c1da118299f228a3e0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209154378
**[Test build #55650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55650/consoleFull)** for PR 12334 at commit [`944e8ab`](https://github.com/apache/spark/commit/944e8ab3da77a406fec9a8c1da118299f228a3e0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209148784
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55647/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209155625
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59596071
--- Diff: pom.xml ---
@@ -274,6 +274,11 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>com.esotericsoftware</groupId>
--- End diff --
Note that this isn't actually affecting the Kryo version, since the `dev/dependencies` files weren't updated. The motivation for this was that `unsafe` was depending on `chill`, a Scala dependency, when in fact it only needed Kryo. To remove this Chill dependency so that `unsafe` could be a pure-Java module, I replaced it by a direct dependency on Kryo and used the exact same dependency and version that Chill was pulling in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209258675
Overall looks ok, just have a few minor questions:
- now there's a single, unversioned "spark-parent" regardless of which Scala version you're using; did you make sure that compiling an app against Spark/2.10 using a spark-parent pom published by the 2.11 build doesn't inadvertently cause any 2.11-specific things to leak out?
- similarly, I assume there will be no issues while deploying artifacts to maven, since the 2.10 and 2.11 builds will now generate duplicate artifacts.
- regarding not supporting cross builds for Spark, is that really something we don't want to support? I've hacked together something (non-Spark-related) in maven to allow cross-compiling in the same build, and I think that if we're going to support multiple versions of Scala, that might be a good thing to have in Spark; it's not unusual for the 2.10 build to be broken these days because it's not exercised by the PR builders.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen closed the pull request at:
https://github.com/apache/spark/pull/12334
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12334#discussion_r59612645
--- Diff: assembly/pom.xml ---
@@ -20,7 +20,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.spark</groupId>
- <artifactId>spark-parent_2.11</artifactId>
+ <artifactId>spark-parent</artifactId>
--- End diff --
It looks like there's a more complete summary of Maven's desired behavior at https://cwiki.apache.org/confluence/display/MAVENOLD/Artifact-Coordinate+Expression+Transformation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209148782
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14574][BUILD][test-maven] Stop cross-bu...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12334#issuecomment-209200571
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org