You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by witgo <gi...@git.apache.org> on 2014/04/22 09:36:01 UTC
[GitHub] spark pull request: Improved build configuration
GitHub user witgo opened a pull request:
https://github.com/apache/spark/pull/480
Improved build configuration
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/witgo/spark format_pom
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/480.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #480
----
commit 0c6c1fc4c6005def6391beadd55e463cc5b65344
Author: witgo <wi...@qq.com>
Date: 2014-04-22T07:29:53Z
Fix compile spark core error with hadoop 0.23.x
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11858762
--- Diff: pom.xml ---
@@ -892,10 +1081,11 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
+ <version>3.4.5</version>
--- End diff --
[SPARK-1064](https://issues.apache.org/jira/browse/SPARK-1064),[PR 102](https://github.com/apache/spark/pull/102).There is no equivalent feature in sbt.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11842656
--- Diff: examples/pom.xml ---
@@ -124,6 +110,10 @@
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
+ <exclusion>
--- End diff --
This might be a good idea but what's the motivation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41640257
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11852495
--- Diff: pom.xml ---
@@ -793,6 +831,157 @@
</build>
<profiles>
+ <!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
+ <profile>
--- End diff --
I do not know how to do, an example?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r12078863
--- Diff: docs/building-with-maven.md ---
@@ -39,6 +39,9 @@ For Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions wit
# Cloudera CDH 4.2.0 with MapReduce v1
$ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package
+ # Apache Hadoop 0.23.x
+ $ mvn -Phadoop-0.23 -Dhadoop.version=0.23.7 -DskipTests clean package
+
For Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN, you should enable the "yarn-alpha" or "yarn" profile and set the "hadoop.version", "yarn.version" property:
--- End diff --
now that we support building without YARN... this should say "you can enable" instead of "you should enable"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11855605
--- Diff: examples/pom.xml ---
@@ -124,6 +110,10 @@
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
+ <exclusion>
--- End diff --
Sounds good. Spark doesn't use JRuby directly and I can't imagine how it uses it indirectly. Then again, this is just in the `examples` module where there is a load of dependency anyway, and which people don't depend on directly. So it doesn't really hurt to not manually prune this stuff. In `core` this would be very important. IMHO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11856833
--- Diff: pom.xml ---
@@ -892,10 +1081,11 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
+ <version>3.4.5</version>
--- End diff --
curator-recipes 2.4.0 => zookeeper 3.4.5
hbase 0.94.6 => zookeeper 3.4.5
kafka_2.10 0.8.0 => zookeeper 3.3.4
hadoop is not directly depend on the zookeeper
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41646101
Cool!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11852713
--- Diff: bagel/pom.xml ---
@@ -31,20 +31,6 @@
<name>Spark Project Bagel</name>
<url>http://spark.apache.org/</url>
- <profiles>
--- End diff --
It exists in almost all modules should be referred to the parent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11914033
--- Diff: pom.xml ---
@@ -506,7 +508,45 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <version>1.7.4</version>
+ <version>${avro.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.jboss.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro-ipc</artifactId>
--- End diff --
Doesn't avro depend on avro-ipc? Why do we need to include both?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11933015
--- Diff: pom.xml ---
@@ -506,7 +508,45 @@
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
- <version>1.7.4</version>
+ <version>${avro.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.jboss.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro-ipc</artifactId>
--- End diff --
spark-hive dependency:
```
[INFO] +- org.apache.hive:hive-serde:jar:0.12.0:compile
[INFO] | +- org.apache.hive:hive-common:jar:0.12.0:compile
[INFO] | | +- org.apache.hive:hive-shims:jar:0.12.0:compile
[INFO] | | | \- commons-logging:commons-logging-api:jar:1.0.4:compile
[INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | +- org.mockito:mockito-all:jar:1.8.5:test (version managed from 1.8.2; scope managed from compile)
[INFO] | +- org.apache.thrift:libfb303:jar:0.9.0:compile
[INFO] | | \- org.apache.thrift:libthrift:jar:0.9.0:compile
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.1.3:compile
[INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.1.3:compile
[INFO] | +- commons-codec:commons-codec:jar:1.4:compile
[INFO] | +- org.apache.avro:avro:jar:1.7.4:compile (version managed from 1.7.1)
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | \- org.apache.avro:avro-mapred:jar:1.7.1:compile
[INFO] | \- org.apache.avro:avro-ipc:jar:1.7.1:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
```
spark-streaming-flume dependency:
```
[INFO] +- org.apache.flume:flume-ng-sdk:jar:1.2.0:compile
[INFO] | +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
[INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
[INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | +- org.apache.avro:avro-ipc:jar:1.6.3:compile
[INFO] | | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | | \- org.apache.velocity:velocity:jar:1.7:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | | \- commons-lang:commons-lang:jar:2.4:compile
```
inconsistent versions dependency
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11842634
--- Diff: pom.xml ---
@@ -793,6 +831,157 @@
</build>
<profiles>
+ <!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
+ <profile>
--- End diff --
I don't think this copy-and-paste approach is reasonable to maintain. Have you tried writing a rule that triggers on a *version range* of the `hadoop-client`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11852992
--- Diff: examples/pom.xml ---
@@ -124,6 +110,10 @@
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
+ <exclusion>
--- End diff --
The jar is very big, there are 12 m.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11852400
--- Diff: pom.xml ---
@@ -519,6 +519,44 @@
</exclusions>
</dependency>
<dependency>
+ <groupId>org.apache.avro</groupId>
--- End diff --
spark-hive dependency:
```
[INFO] +- org.apache.hive:hive-serde:jar:0.12.0:compile
[INFO] | +- org.apache.hive:hive-common:jar:0.12.0:compile
[INFO] | | +- org.apache.hive:hive-shims:jar:0.12.0:compile
[INFO] | | | \- commons-logging:commons-logging-api:jar:1.0.4:compile
[INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | +- org.mockito:mockito-all:jar:1.8.5:test (version managed from 1.8.2; scope managed from compile)
[INFO] | +- org.apache.thrift:libfb303:jar:0.9.0:compile
[INFO] | | \- org.apache.thrift:libthrift:jar:0.9.0:compile
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.1.3:compile
[INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.1.3:compile
[INFO] | +- commons-codec:commons-codec:jar:1.4:compile
[INFO] | +- org.apache.avro:avro:jar:1.7.4:compile (version managed from 1.7.1)
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | \- org.apache.avro:avro-mapred:jar:1.7.1:compile
[INFO] | \- org.apache.avro:avro-ipc:jar:1.7.1:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] | \- org.mortbay.jetty:servlet-api:jar:2.5-20081211:compile
```
spark-streaming-flume dependency:
```
[INFO] +- org.apache.flume:flume-ng-sdk:jar:1.2.0:compile
[INFO] | +- org.apache.avro:avro:jar:1.7.4:compile
[INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
[INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
[INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | +- org.apache.avro:avro-ipc:jar:1.6.3:compile
[INFO] | | +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] | | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] | | \- org.apache.velocity:velocity:jar:1.7:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | | \- commons-lang:commons-lang:jar:2.4:compile
```
inconsistent versions dependency and here to add only affects the module who dependency avro-ipc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41643020
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11933105
--- Diff: pom.xml ---
@@ -793,6 +833,17 @@
</build>
<profiles>
+ <!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
+ <profile>
+ <id>hadoop-0.23</id>
--- End diff --
I have not found this problem in the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11857494
--- Diff: pom.xml ---
@@ -892,10 +1081,11 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
+ <version>3.4.5</version>
--- End diff --
I see, this declaration is just there to make zookeeper "provided" in order to not conflict with a cluster's copy/version of ZK from Hadoop. Hadoop version kind of matters in that sense. I wonder if this is actually best left unspecified then, since there is no intent to enforce any particular version of ZK, just make sure it is not in the artifact when built with this profile?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41644742
I've merged this, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41011689
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11842744
--- Diff: pom.xml ---
@@ -519,6 +519,44 @@
</exclusions>
</dependency>
<dependency>
+ <groupId>org.apache.avro</groupId>
--- End diff --
Same question, what's the need for these new artifacts in all modules?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11913925
--- Diff: pom.xml ---
@@ -793,6 +833,17 @@
</build>
<profiles>
+ <!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
+ <profile>
+ <id>hadoop-0.23</id>
--- End diff --
I think in the past we had trouble defining this profile in the parent pom. The issue was that profiles were not correctly inherited by the sub projects, because profiles are not transative. I don't remember how exactly this manifested though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/480
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41640359
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11842700
--- Diff: bagel/pom.xml ---
@@ -31,20 +31,6 @@
<name>Spark Project Bagel</name>
<url>http://spark.apache.org/</url>
- <profiles>
--- End diff --
I see what you're doing, to lift these dependencies up into the parent, but they will then be applied to *all* modules. Is that desirable -- which modules now have this dependency that didn't before, or are there none?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41640367
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11855383
--- Diff: pom.xml ---
@@ -793,6 +831,157 @@
</build>
<profiles>
+ <!-- SPARK-1121: Adds an explicit dependency on Avro to work around a Hadoop 0.23.X issue -->
+ <profile>
--- End diff --
Scratch that. I tried a bunch of variations on this and can't find a way to activate the profile based on a range. I think it's best to activate this profile explicitly with `-P` rather than maintain all this duplication, still.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41641309
So I looked through this a bunch and ran several builds with different permutations. This looks really good! Thanks @witgo I'm sure it took some time to figure all this out. I'm going to merge this pending tests because I'd like to cut an RC tonight and I don't want to change the build around after we ask people to test stuff.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Improved build configuration
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/480#issuecomment-41643021
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14552/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11855786
--- Diff: pom.xml ---
@@ -519,6 +519,44 @@
</exclusions>
</dependency>
<dependency>
+ <groupId>org.apache.avro</groupId>
--- End diff --
I see, yes, +1 I would introduce an `avro.version` property to set all of these consistently to 1.7.4 in one place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [WIP]Improved build configuration
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/480#discussion_r11855871
--- Diff: pom.xml ---
@@ -892,10 +1081,11 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
+ <version>3.4.5</version>
--- End diff --
+1, this looks like it was unspecified. Does this version need to vary with Hadoop version? And also in the SBT build?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---