You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2016/03/31 00:27:13 UTC
[GitHub] spark pull request: [SPARK-11416][BUILD Update to Chill 0.8.0 & Kr...
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/12076
[SPARK-11416][BUILD Update to Chill 0.8.0 & Kryo 3.0.3
Thisw
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark kryo3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12076.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12076
----
commit cb8a040c657f3917d6573d102fa96688914ac0c0
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-03-30T22:25:51Z
[SPARK-11416] Update to Chill 0.8.0 & Kryo 3.0.3
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-204325755
...Thinking about this; it might be possible to go to hive with a shaded kryo, with the invocation of those methods which exchange kryo types referring to the shaded values. That doesn't do anything for marshalling stuff between kryo versions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207642269
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203693449
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207608141
**[Test build #55393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55393/consoleFull)** for PR 12076 at commit [`4fc0871`](https://github.com/apache/spark/commit/4fc0871a3e0b3562fa715137c5f5df1fcb77b5da).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207126856
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55257/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203722178
**[Test build #54571 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54571/consoleFull)** for PR 12076 at commit [`cb8a040`](https://github.com/apache/spark/commit/cb8a040c657f3917d6573d102fa96688914ac0c0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203696314
Test failures appear unrelated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207644614
Alright, I'm going to go ahead and merge this. We'll upgrade to 2.12-friendly versions of Chill + Kryo as part of the 2.12 testing patch which will come out later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207124264
**[Test build #55257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55257/consoleFull)** for PR 12076 at commit [`449f581`](https://github.com/apache/spark/commit/449f58117a88cf50a055742d85ea668771d9236b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207642274
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55393/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-206514828
Splitting the Hive Kryo shading into a new JIRA: https://issues.apache.org/jira/browse/SPARK-14435
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203677962
Seemingly none that affect us so far. There are some binary incompatibilities in `KryoPool` (it became an interface), but that's not going to impact Spark itself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203668279
**[Test build #54558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54558/consoleFull)** for PR 12076 at commit [`cb8a040`](https://github.com/apache/spark/commit/cb8a040c657f3917d6573d102fa96688914ac0c0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203722281
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-205690484
My understanding so far:
- [Marcelo showed](https://github.com/apache/spark/pull/12076#issuecomment-204129446) that we can use shaded Kryo in Hive.
- While the Protobuf shading might have originally been introduced for Hadoop 1.x support, it's probably a good idea to keep the shading because Protobuf is a conflict-prone dependency.
- We can't use the regular `hive-exec` artifacts because they package tons of dependencies without relocation.
- We can't use the `core`-classified `hive-exec` 1.2.1 artifacts because they don't shade the things that we need shaded. Even if we could use this artifact, it is a pain to consume because its POM does not declare required dependencies, so we have to manually add all of Hive's transitive deps as direct deps. We've already sort of done this, but I believe that it's unnecessary and am in the process of undoing a lot of those changes as part of #1217.
Therefore, I think there's no avoiding having to publish another custom Hive 1.2.1 build. I propose that we do so by editing the 1.2.1-spark POM to restore Kryo shading. I'll work on this tomorrow and loop back to this discussion once I've completed the required Hive dep. bumps.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203697517
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207126853
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-204324891
1. Hive uses Kryo "the guava of serialization" internally; I don't know the specifics, but its not insignificant.
1. they moved ahead of spark's version to fix some bugs; that's the kryo which repackaged things.
1. Kryo types were used across methods called from Spark; the unshading was needed to get things to link, same with the switch from 2.22 to 2.21 (and a change to some files). It was less traumatic to push Hive back slightly than to try to force an update into unknown territory for chill & dependencies, even if it were possible.
Upgrading to hive 2 would finally get everything in sync: ideally eliminate the need to have a custom hive JAR at all. Essentially all spark needs is a version of Hive with parameters that can be exchanged across all needed methods, with the conflict packages shaded, and the non-conflict packages generally omitted
If you want to find out where things don't link, edit the spark pom to pull in org.apache.hive/hive and see what breaks.
(of course, the other thing that would be nice would be for Hive to make it possible to subclass their thrift service cleanly; I've got a PoC of that somewhere ... it's not hard to add)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-204588620
I don't recall doing any reflection related stuff to work with kryo; problems were showing up in compilation.
There's one more thing to worry about, hive minimal vs uber jars. It may still be necessary to have a custom 1.2.x JAR, just to keep all the other cruft out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-204503015
Grepping `sql/hive` for `kryo` turns up one explicit usage in HiveShim's `HivefunctionWrapper` where it looks like we take an instance of `Kryo` provided by Hive and use that instance to deserialize things: https://github.com/apache/spark/blob/8ba2b7f28fee39c4839e5ea125bd25f5091a3a1e/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala#L171. Assuming that we use a shaded Kryo in Hive, I think we could just import the shaded Kryo here and use that import in our `[de]serializeObjectByKryo` methods' signatures.
I don't spot any other Hive-specific uses of Kryo which look problematic, so it seems plausible that we can shade without major problems. Am I overlooking other uses of Kryo to exchange data between Spark and Hive internals?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by johnynek <gi...@git.apache.org>.
Github user johnynek commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-204045391
quick glance, it looked to me like the failures were hive related. I don't know if you use any hive code, but could the fact that hive also uses kryo create a diamond that gives a runtime failure?
Just a shot in the dark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207642014
**[Test build #55393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55393/consoleFull)** for PR 12076 at commit [`4fc0871`](https://github.com/apache/spark/commit/4fc0871a3e0b3562fa715137c5f5df1fcb77b5da).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12076#discussion_r57975046
--- Diff: dev/deps/spark-deps-hadoop-2.2 ---
@@ -122,7 +122,7 @@ jsr305-1.3.9.jar
jta-1.1.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.16.jar
-kryo-2.21.jar
+kryo-shaded-3.0.3.jar
--- End diff --
Thanks to this shading, we no longer need to add exclusions for Kryo's ASM dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207134384
**[Test build #55265 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55265/consoleFull)** for PR 12076 at commit [`b658b3b`](https://github.com/apache/spark/commit/b658b3bacd0b8de1442ceca27d7378d329149095).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207163692
**[Test build #55265 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55265/consoleFull)** for PR 12076 at commit [`b658b3b`](https://github.com/apache/spark/commit/b658b3bacd0b8de1442ceca27d7378d329149095).
* This patch **fails Spark unit tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12076
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207164162
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203722283
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54571/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207126835
**[Test build #55257 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55257/consoleFull)** for PR 12076 at commit [`449f581`](https://github.com/apache/spark/commit/449f58117a88cf50a055742d85ea668771d9236b).
* This patch **fails build dependency tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203697881
**[Test build #54571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54571/consoleFull)** for PR 12076 at commit [`cb8a040`](https://github.com/apache/spark/commit/cb8a040c657f3917d6573d102fa96688914ac0c0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203693453
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54558/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-205739991
Makes sense. Note that getting the SBT dependencies to match maven's was a complete nightmare; it'd probably have been easier to write a new version reconciler for ivy than solve it for spark alone. You shouldn't have to do much to that listing as it stands today, but updating to later hive versions will probably hurt again.
FWIW, Hadoop 2.x isn't going to increment protobuf, I'm confident of that to the extent that there are enough people who would veto any attempt. Worry more about guava and jackson
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203693292
**[Test build #54558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54558/consoleFull)** for PR 12076 at commit [`cb8a040`](https://github.com/apache/spark/commit/cb8a040c657f3917d6573d102fa96688914ac0c0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203731423
Huh, weird. It looks like this somehow _is_ a legitimate failure. I'll investigate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207608301
The Hive conflict should now be fixed now that #12215 has been merged, so I've updated this and am planning to merge as soon as tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-207164166
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55265/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-203676999
No API change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Posted by magro <gi...@git.apache.org>.
Github user magro commented on the pull request:
https://github.com/apache/spark/pull/12076#issuecomment-204069349
Btw, we'll have a kryo 3.1.0 release soon (depends on pending fixes regarding handling of generics). This will contain an incompatibility regarding Closure handling (https://github.com/EsotericSoftware/kryo/pull/415), not sure if that's relevant for you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org