You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by avati <gi...@git.apache.org> on 2014/07/30 07:22:24 UTC
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
GitHub user avati opened a pull request:
https://github.com/apache/spark/pull/1649
[SPARK-1812] [WIP] Scala 2.11 support
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/avati/spark scala-2.11
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1649.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1649
----
commit f8b5e96fca20c13308cb2a9a6c18049bcdd0a7ba
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-30T05:12:30Z
SPARK-1812: use consistent protobuf version across build
Moving to akka-2.3 reveals issues where differing protobuf
versions within the same build results in runtime incompatibility
and exceptions like:
java.lang.VerifyError: class akka.remote.WireFormats$AkkaControlMessage overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
Signed-off-by: Anand Avati <av...@redhat.com>
commit c6e0c6ea3b0f5b3316f01f1a743f4de0a6cdf816
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T07:48:49Z
SPARK-1812: sql/catalyst - upgrade to scala-logging-slf4j 2.1.2
Signed-off-by: Anand Avati <av...@redhat.com>
commit 722aee26399b9bf4b725d17f5cfcfad99464af35
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-27T02:20:34Z
SPARK-1812: core - upgrade to akka 2.3.4
Signed-off-by: Anand Avati <av...@redhat.com>
commit 0d3e37df46c6ce03fe94e10ec26d5bcb96ee86c5
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-27T03:39:15Z
SPARK-1812: core - upgrade to json4s 3.2.10
Signed-off-by: Anand Avati <av...@redhat.com>
commit 1d2600d41caf6bc47977a5aee9ded7a7aaf85818
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-27T03:43:27Z
SPARK-1812: core - upgrade to unofficial chill 1.2 (tv.cntt)
Signed-off-by: Anand Avati <av...@redhat.com>
commit 0d44a59234f94246dd414ef136647fc4483a9180
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T00:19:17Z
SPARK-1812: core - Fix overloaded methods with default arguments
Signed-off-by: Anand Avati <av...@redhat.com>
commit 96f449416a13cba033ff17806a9c56e7aea2692f
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T00:21:39Z
SPARK-1812: Fix @transient annotation errors
- convert to @transient val when unsure
- remove @transient annotation elsewhere
Signed-off-by: Anand Avati <av...@redhat.com>
commit d39f8678fd4b2aff96d7d6fe15b2878ef085ce0b
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T04:06:48Z
SPARK-1812: mllib - upgrade to breeze 0.8.1
Signed-off-by: Anand Avati <av...@redhat.com>
commit d7cf8c8c0476d3c6e7adcea434bb87a2aa665bcf
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T04:16:52Z
SPARK-1812: streaming - Fix overloaded methods with default arguments
Signed-off-by: Anand Avati <av...@redhat.com>
commit caef846be53f83498b0603e569404389c792c0b1
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T00:20:03Z
SPARK-1812: core - [FIXWARNING] try { } without catch { } warning
Signed-off-by: Anand Avati <av...@redhat.com>
commit 0f68c91d6001f3db516a2e9d6d936d65122bf997
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T08:02:49Z
SPARK-1812: temporarily disable
- remove spark-repl from assembly dependency
- temporarily disable external/kafka module (dependency unavailable)
- examples module
- kafka_2.11 (transitive)
- algebird_2.11 (com.twitter)
Signed-off-by: Anand Avati <av...@redhat.com>
commit c4f92b6f9bb798dafae69ad8432292be8c3a58d4
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T08:15:22Z
SPARK-1812: move to akka-zeromq-2.11.0-M3 version 2.2.0
.. until akka-zeromq-2.11 gets published
Signed-off-by: Anand Avati <av...@redhat.com>
commit 7f4d34bf499d2d238925ca777c6f6a78a31904af
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T23:35:51Z
SPARK-1812: core - [FIXWARNING] replace scala.concurrent.{future,promise} with {Future,Promise}
Signed-off-by: Anand Avati <av...@redhat.com>
commit c50ff00c97d3de41576a7d61ee4d214cafc9b089
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-26T08:01:32Z
SPARK-1812: update all artifactId to spark-${module}_2.11
Signed-off-by: Anand Avati <av...@redhat.com>
commit 8e621813d77266a51a09d4a217352d61b7e1861e
Author: Anand Avati <av...@redhat.com>
Date: 2014-07-27T04:33:42Z
SPARK-1812: sql/core - [FIXWARNING] replace scala.concurrent.future with Future
Signed-off-by: Anand Avati <av...@redhat.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by avati <gi...@git.apache.org>.
Github user avati commented on a diff in the pull request:
https://github.com/apache/spark/pull/1649#discussion_r15598196
--- Diff: pom.xml ---
@@ -1015,7 +1016,6 @@
<id>hadoop-2.2</id>
<properties>
<hadoop.version>2.2.0</hadoop.version>
- <protobuf.version>2.5.0</protobuf.version>
--- End diff --
I was not aware of Hadoop-1 specific dependency on 2.4.1. Going by what you are saying, my guess is this change will not work with Hadoop 1. I am only testing with "mvn test" within Spark as of now.
So someone has to release a shaded protobuf version of either Hadoop1 or akka to reconcile this problem. That would be necessary before final merge. In the mean time I shall continue to make progress ignoring the Hadoop-1 compatibility problem (after all bulk of the work is yet to be started, in repl). Once a shaded-protobuf is released (akka or hadoop1, whichever) it is just a trivial change in pom.xml then.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by ScrapCodes <gi...@git.apache.org>.
Github user ScrapCodes commented on the pull request:
https://github.com/apache/spark/pull/1649#issuecomment-50718152
Hey Anand,
This looks like a very nice effort !, It would be very convenient if you can break this pull request into smaller subtasks.
For example upgrading akka can be one subtask, and so on. That way it is easier for us to review/test and then merge. If however you think that you will not have the time for that I am happy to do it for you ? We would like to have this pretty soon - breaking up the PR will make reviewing and testing really easy for us.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1649#issuecomment-50643077
On Wed, Jul 30, 2014 at 3:14 AM, Sean Owen <no...@github.com> wrote:
> I wonder if you can pull out the changes that will also work in Scala
> 2.10? some of it looks like some syntactic cleanup which could go in
> regardless of the Scala version? or is some of that only support in 2.11?
>
>
>
The warnings fixed in this PR show up only in 2.11 scala. 2.10 scala build
is warning free. The patches in this PR are necessary for Scala 2.11, but
not exclusive to 2.11. However, because the patches are dependent on each
other, I have committed them all in the dependent order in the same branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by avati <gi...@git.apache.org>.
Github user avati commented on a diff in the pull request:
https://github.com/apache/spark/pull/1649#discussion_r15595520
--- Diff: pom.xml ---
@@ -1015,7 +1016,6 @@
<id>hadoop-2.2</id>
<properties>
<hadoop.version>2.2.0</hadoop.version>
- <protobuf.version>2.5.0</protobuf.version>
--- End diff --
On Wed, Jul 30, 2014 at 3:12 AM, Sean Owen <no...@github.com> wrote:
> In pom.xml:
>
> > @@ -1015,7 +1016,6 @@
> > <id>hadoop-2.2</id>
> > <properties>
> > <hadoop.version>2.2.0</hadoop.version>
> > - <protobuf.version>2.5.0</protobuf.version>
>
> Why would removing this be related to Scala 2.11? These are important
>
>
>
My goal was to have a consistent version of protobuf be used across the
build. As I found (and by others, e.g
https://groups.google.com/forum/#!topic/akka-user/cI4CEKEJvfs) the
mismatching protobuf version is causing akka-2.3 to not work. Indeed, using
a single $protobuf.version in pom.xml did indeed fix the akka timeout bug
(described in more detail in Jira 1812). And akka-2.3 is needed because
that is the lowest akka version relased in Scala 2.11.
I could modify this patch to just change 2.4.1 to 2.5.0 in the top pom.xml
profile and leave out the hadoop accompanying versions as-is. Would you
prefer it that way?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1649#issuecomment-50575027
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by avati <gi...@git.apache.org>.
Github user avati closed the pull request at:
https://github.com/apache/spark/pull/1649
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by avati <gi...@git.apache.org>.
Github user avati commented on a diff in the pull request:
https://github.com/apache/spark/pull/1649#discussion_r15596377
--- Diff: pom.xml ---
@@ -1015,7 +1016,6 @@
<id>hadoop-2.2</id>
<properties>
<hadoop.version>2.2.0</hadoop.version>
- <protobuf.version>2.5.0</protobuf.version>
--- End diff --
I'm not sure about why protobuf was shaded to begin with, because non-shaded akka-2.2 works just fine (shaded version was akka-2.2). Only akka-2.3 (recent version) caused the timeouts because of protobuf mismatch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/1649#issuecomment-50597071
I wonder if you can pull out the changes that will also work in Scala 2.10? some of it looks like some syntactic cleanup which could go in regardless of the Scala version? or is some of that only support in 2.11?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1649#discussion_r15595677
--- Diff: pom.xml ---
@@ -1015,7 +1016,6 @@
<id>hadoop-2.2</id>
<properties>
<hadoop.version>2.2.0</hadoop.version>
- <protobuf.version>2.5.0</protobuf.version>
--- End diff --
I think this would then cause the whole thing to be incompatible with Hadoop 1.x. The akka version would have to go with the protobuf version of Hadoop, so the newer version could only work with Hadoop 2.x+. I think this is why akka's protobuf was shaded to begin with? Changing it to 2.5.0 globally would again break Hadoop 1.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1649#discussion_r15575515
--- Diff: pom.xml ---
@@ -1015,7 +1016,6 @@
<id>hadoop-2.2</id>
<properties>
<hadoop.version>2.2.0</hadoop.version>
- <protobuf.version>2.5.0</protobuf.version>
--- End diff --
Why would removing this be related to Scala 2.11? These are important
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1649#issuecomment-50720340
I will split this up into multiple PRs.
I will start with a small pull request for akka upgrade to 2.3. However as Sean mentioned, I guess we are stuck on the fundamental incompatibility with Hadoop 1. Either we will need to forego Hadoop 1 compatibility, or wait till a -shaded-protobuf version of akka artifact is released (like akka-2.2.3-shaded-protobuf from org.spark-project.akka), or temporarily forego compatibility and fixup pom.xml later if/when a shaded-protobuf artifact is released.
I would like to hear opinions about this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1649#discussion_r15575545
--- Diff: pom.xml ---
@@ -111,16 +111,16 @@
<java.version>1.6</java.version>
<sbt.project.name>spark</sbt.project.name>
- <scala.version>2.10.4</scala.version>
- <scala.binary.version>2.10</scala.binary.version>
+ <scala.version>2.11.1</scala.version>
+ <scala.binary.version>2.11</scala.binary.version>
<mesos.version>0.18.1</mesos.version>
<mesos.classifier>shaded-protobuf</mesos.classifier>
- <akka.group>org.spark-project.akka</akka.group>
- <akka.version>2.2.3-shaded-protobuf</akka.version>
+ <akka.group>com.typesafe.akka</akka.group>
+ <akka.version>2.3.4</akka.version>
<slf4j.version>1.7.5</slf4j.version>
<log4j.version>1.2.17</log4j.version>
<hadoop.version>1.0.4</hadoop.version>
- <protobuf.version>2.4.1</protobuf.version>
+ <protobuf.version>2.5.0</protobuf.version>
--- End diff --
Same, you can't change this for the default of Hadoop 1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by avati <gi...@git.apache.org>.
Github user avati commented on the pull request:
https://github.com/apache/spark/pull/1649#issuecomment-50833567
Splitting into multiple smaller PRs. Closing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1812] [WIP] Scala 2.11 support
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/1649#discussion_r15596775
--- Diff: pom.xml ---
@@ -1015,7 +1016,6 @@
<id>hadoop-2.2</id>
<properties>
<hadoop.version>2.2.0</hadoop.version>
- <protobuf.version>2.5.0</protobuf.version>
--- End diff --
But did it work with Hadoop 1? that needs protobuf 2.4.1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---