You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JoshRosen <gi...@git.apache.org> on 2016/04/28 23:33:10 UTC
[GitHub] spark pull request: [SPARK-14989] Upgrade Jackson from 2.5.3 to 2....
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/12766
[SPARK-14989] Upgrade Jackson from 2.5.3 to 2.7.3
This patch upgrades Jackson from 2.5.3 to 2.7.3. I'd like to upgrade now in order to take advantage of new performance improvements and features, as well as to be better-prepared for when we'll want to upgrade for Scala 2.12 support.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark upgrade-jackson
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12766.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12766
----
commit 264f76fafca7ea4909f4205e3c8a6a440b39a055
Author: Josh Rosen <jo...@databricks.com>
Date: 2016-04-28T21:28:05Z
Upgrade Jackson from 2.5.3 to 2.7.3
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215591871
**[Test build #57282 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57282/consoleFull)** for PR 12766 at commit [`264f76f`](https://github.com/apache/spark/commit/264f76fafca7ea4909f4205e3c8a6a440b39a055).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215596199
/cc @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215659662
Actually may be fine. Hadoop actually includes Jackson 1.x, and the things pulling in 2.x besides Spark seem to be dropwizard metrics and Scala. Those may be much more manageable
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215592028
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215592032
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57282/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215657834
I like it. The only looming problem is whether this works in the context of Hadoop classes which may have a different Jackson version. As long as 2.7 wins, it probably works, but it bears testing. Shading could help if there's a problem. This dependency is always a little problematic but worth pushing forward.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-222197705
Closing this PR for now since this upgrade was performed in a different PR while I was away.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215570639
**[Test build #57282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57282/consoleFull)** for PR 12766 at commit [`264f76f`](https://github.com/apache/spark/commit/264f76fafca7ea4909f4205e3c8a6a440b39a055).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by cowtowncoder <gi...@git.apache.org>.
Github user cowtowncoder commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215876639
@JoshRosen yes, thank you for following up on this!
On compatibility: there were some issues with 2.7.0 - 2.7.2, regarding type resolution, most of which were fixed with 2.7.3.
With 2.7.4 (released today) I hope that regressions are finally dealt with; there were some more esoteric edge cases.
2.6.x had one compatibility issue over 2.5.x that users have had problems with (wrt `JsonInclude.Include.NON_EMPTY` applying to default values of primitive/wrapper types), so it may be safest to skip that version (2.5.5 is latest from 2.5). Behavior of 2.7 is same as that of 2.5; 2.7 added better support for option `JsonInclude.Include.NON_DEFAULT` to allow suppression of values like `0` for `int`, but keeping `NON_EMPTY` reserved for container things (and empty `String`).
This is the only major compatibility concern I am aware of.
So with Spark 2.0, I would recommend going with 2.7.4.
One additional thing that may make 2.7 best choice is that we once again have active owner for Scala module, and there has been progress in getting some long-time-open bugs fixed for 2.7.4.
And there is also Scala 2.12 variant of that module now (starting with 2.7.3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/12766#discussion_r61546894
--- Diff: dev/deps/spark-deps-hadoop-2.7 ---
@@ -141,7 +142,7 @@ netty-all-4.0.29.Final.jar
objenesis-2.1.jar
opencsv-2.3.jar
oro-2.0.8.jar
-paranamer-2.6.jar
+paranamer-2.3.jar
--- End diff --
Weird that this version went down
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen closed the pull request at:
https://github.com/apache/spark/pull/12766
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14989][BUILD] Upgrade Jackson from 2.5....
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/12766#issuecomment-215850951
My impression is that Jackson's backwards-compatibility story is good.
It looks like Jackson 2.7.4 was just released a few hours ago, so maybe we should wait a day and upgrade straight to that instead: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.7.4
@cowtowncoder, since you [mentioned](https://twitter.com/cowtowncoder/status/725737598044393472) wanting to get the various "big data" frameworks to upgrade to newer Jackson versions, do you know of any compatibility issues that we should be aware of? Just thought I'd ask since you might have received feedback from other projects that are performing the same upgrade.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org