You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2018/08/14 03:47:00 UTC
[jira] [Comment Edited] (SPARK-24771) Upgrade AVRO version from
1.7.7 to 1.8
[ https://issues.apache.org/jira/browse/SPARK-24771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579213#comment-16579213 ]
Marcelo Vanzin edited comment on SPARK-24771 at 8/14/18 3:46 AM:
-----------------------------------------------------------------
The main problem pointed out in the original attempt is AVRO-1502; it means that people with code generated by Avro 1.7 might run into problems if Spark ships with Avro 1.8; that basically amounts to a binary compatibility issue, which we always try not to break in minor releases.
It may be that it only applies in some specific situations and it may be acceptable to release note it. But it would be nice to be sure, since this affects existing users of Avro - including the Flume streaming connector which is still available in Spark 2.4...
(Edit: the flume connector re-compiles the avro interface during the Spark build, but it would be nice to check that the flume libraries themselves don't use avro for anything. And we still should check how much other users of avro would be affected - e.g. people who use avro directly in RDD code.)
was (Author: vanzin):
The main problem pointed out in the original attempt is AVRO-1502; it means that people with code generated by Avro 1.7 might run into problems if Spark ships with Avro 1.8; that basically amounts to a binary compatibility issue, which we always try not to break in minor releases.
It may be that it only applies in some specific situations and it may be acceptable to release note it. But it would be nice to be sure, since this affects existing users of Avro - including the Flume streaming connector which is still available in Spark 2.4...
> Upgrade AVRO version from 1.7.7 to 1.8
> --------------------------------------
>
> Key: SPARK-24771
> URL: https://issues.apache.org/jira/browse/SPARK-24771
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Gengliang Wang
> Assignee: Gengliang Wang
> Priority: Major
> Fix For: 2.4.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org