You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by benmccann <gi...@git.apache.org> on 2016/06/30 01:57:36 UTC
[GitHub] spark pull request #13986: Upgrade to Avro 1.8.1
GitHub user benmccann opened a pull request:
https://github.com/apache/spark/pull/13986
Upgrade to Avro 1.8.1
## What changes were proposed in this pull request?
Upgrade to Avro 1.8.1
## How was this patch tested?
Ran `mvn test`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/benmccann/spark avro-1.8.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13986.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13986
----
commit 9ca14a76d0b0dcf2a6d7f8cf39667faf585817bd
Author: Ben McCann <bm...@linkedin.com>
Date: 2016-06-30T00:54:03Z
Upgrade to Avro 1.8.1
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13986: Upgrade to Avro 1.8.1
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/13986
FYI, it seems its 1.7.4 for Hadoop 2.7, https://github.com/apache/hadoop/blob/release-2.7.0/hadoop-project/pom.xml#L60-L61 and Hadoop 2.2, https://github.com/apache/hadoop/blob/release-2.2.0/hadoop-project/pom.xml#L62-L63.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13986: Upgrade to Avro 1.8.1
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/13986
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #13986: [SPARK-16617] Upgrade to Avro 1.8.1
Posted by benmccann <gi...@git.apache.org>.
Github user benmccann closed the pull request at:
https://github.com/apache/spark/pull/13986
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13986: Upgrade to Avro 1.8.1
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13986
We probably don't want to mismatch Hadoop's version unless you've vetted this fairly carefully that it is backwards compatible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13986: Upgrade to Avro 1.8.1
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13986
In general that has a lot of implications, since I think this will mismatch the version in several Hadoop versions. You probably need to
- Read https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
- Fix up this PR accordingly
- At least run the script that updates the deps/ files to see that impact
- Check what version Hadoop 2.2 and 2.7 use as references
- Figure out what the upside is in updating to 1.8
- Note any potential incompatibilities here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13986: [SPARK-16617] Upgrade to Avro 1.8.1
Posted by benmccann <gi...@git.apache.org>.
Github user benmccann commented on the issue:
https://github.com/apache/spark/pull/13986
>At least run the script that updates the deps/ files to see that impact
Can you point me to the script you're referring to?
> Figure out what the upside is in updating to 1.8
The reason I'm proposing this change is that Avro 1.8 makes the generated classes serializable so that you can create RDDs containing Avro objects.
> Note any potential incompatibilities here
Here's the list of incompatible changes from the [changelog](https://github.com/apache/avro/blob/master/CHANGES.txt):
* AVRO-1334. Java: Update versions of many dependencies. (scottcarey, cutting)
* AVRO-997. Java: For enum values, no longer sometimes permit any Object whose toString() names an enum symbol, but rather always require use of distinct enum types. (Sean Busbey via cutting)
* AVRO-1602. Java: Remove Dapper-style RPC trace facility. This seems unused and has been a source of build problems. (cutting)
* AVRO-1586. Build against Hadoop 2. With this change the avro-mapred and trevni-avro JARs without a hadoop1 or hadoop2 Maven classifier are Hadoop 2 artifacts. To use with Hadoop 1, set the classifier to hadoop1. (tomwhite)
* AVRO-1502. Java: Generated classes now implement Serializable. Generated classes need to be regenerated to use this release. (cutting)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13986: [SPARK-16617] Upgrade to Avro 1.8.1
Posted by benmccann <gi...@git.apache.org>.
Github user benmccann commented on the issue:
https://github.com/apache/spark/pull/13986
I'll close for now until Hadoop 3.x. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #13986: [SPARK-16617] Upgrade to Avro 1.8.1
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/13986
Have a look at `dev/test-dependencies.sh --replace-manifest`. I think the big concern is matching the Hadoop dependency, which will be on 1.7.x for 2.x. Updating to the latest 1.7.x seems OK. You can also test this change anyway after making the deps change to see what happens.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org