You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2019/07/16 22:57:00 UTC
[jira] [Comment Edited] (SPARK-27416) UnsafeMapData &
UnsafeArrayData Kryo serialization breaks when two machines have different
Oops size
[ https://issues.apache.org/jira/browse/SPARK-27416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886535#comment-16886535 ]
Josh Rosen edited comment on SPARK-27416 at 7/16/19 10:56 PM:
--------------------------------------------------------------
I think that we should backport this for Spark 2.4.4 because the related SPARK-27406 was backported for Spark 2.4.3 and both bugs seem like potential correctness issues (they'll often result in crashes but might theoretically result in wrong results in some cases, something that we saw happen with the original UnsafeRow OOPs bug).
I spotted this while going through the full list of 2.4.0 -> 2.4.3 commits: I saw that the related ArrayData PR was backported but this followup for Maps was not.
I can open a backport PR later; if anyone else wants to pick this up then please feel free to.
/cc [~smilegator]
was (Author: joshrosen):
I think that we should backport this for Spark 2.4.4 because the related SPARK-27406 was backported for Spark 2.4.3 and both bugs seem like potential correctness issues (they'll often result in crashes but might theoretically result in wrong results in some cases, something that we saw happen with the original UnsafeRow OOPs bug).
I spotted this while going through the full list of 2.4.0 -> 2.4.3 commits.
I can open a backport PR later; if anyone else wants to pick this up then please feel free to.
/cc [~smilegator]
> UnsafeMapData & UnsafeArrayData Kryo serialization breaks when two machines have different Oops size
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-27416
> URL: https://issues.apache.org/jira/browse/SPARK-27416
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.1
> Reporter: peng bo
> Assignee: peng bo
> Priority: Major
> Fix For: 3.0.0
>
>
> Actually this's follow up for https://issues.apache.org/jira/browse/SPARK-27406, https://issues.apache.org/jira/browse/SPARK-10914
> This issue is to fix the UnsafeMapData & UnsafeArrayData Kryo serialization issue when two machines have different Oops size.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org